Tag Archives: development

Project versus Product

With the fuss about the Log4Shell vulnerability finally dying down, it’s time to step back and take a good, long think about what happened and, more importantly, what can be done to stop it from happening again.

Sadly the prognosis is not good. The tl;dr is both simple and obvious: we simultaneously like free stuff and getting paid for our own work.

Most companies treat open source software exactly the same as commercial software but with a much lower purchase cost. When the software goes wrong, we want someone else to fix it for us. Unfortunately, sometimes we don’t even know where the software comes from. In the case of log4j, it’s run by volunteers. There is no 24/7 help desk with eager employees waiting to take your call.

But even if there is a company that backs the project, one that does have engineering, QA and support staff, is it reasonable to expect an immediate fix to a vulnerability?

Whether a company backs it or not, using open source software is more like being part of a community than being a customer. I came across this phrase about free software a few years ago: “If it breaks, then you get to keep both pieces.” It’s very apt here.

My small part in trying to fix this perception is by calling free software projects and commercially supported versions products or services.

The idea here is that the word “project” implies some degree of a work in progress, one that requires effort from all stakeholders.

In my Day Job1, I often see companies expecting to get commercial quality support for free because the software is free. By “commercial quality” I don’t mean “good.” The level of knowledge and support provided by most free projects is phenomenal. Instead, I mean that there are service-level agreements and guarantees of service within a particular time frame.

As it says in “Cloud Without Compromise”:

But there is something to the old adage that “You get what you pay for.” (… there’s a world of difference between building systems for pet projects versus designing for the needs of enterprise.)

Those service level agreements come with a cost. If you want a fix or an enhancement, you’re welcome to ask a project for it, but it might never come or, if it does, maybe not on a timescale that helps you. The rules change when you pay for the help.

My hope is that those companies that only ever take from open source projects and never contribute learned a lesson. By helping keep the project healthy, you don’t even need to be altruistic. Think of it as insurance.

If you use the software, there are many ways you can “pay it forward”: share enhancements and fixes, write documentation, share your knowledge and experience to bring in more users, or help other users with community support. But if there’s a mechanism to pay for it, the simplest way for most users is cold, hard cash.

  1. An open source database, so I have both skin in the game and bias. However, I think I’d be saying the same thing even if I worked for a vendor of closed source software. ↩︎

Is git too hard

I stumbled across “On git and cognitive load” and it got me thinking. That post led me to “Oh shit, git!?!” and that got me thinking further.

But first, a disclaimer: this is a post more about having perspective than providing answers. If I knew a better way, I’d be doing it.

The first blog argues that git is difficult to use. Further, that it was designed with the limitations of the time and that those limitations are no longer valid constraints. A decentralised system was required when poor network connections were the norm. Now we have cloud providers and ridiculous amounts of bandwidth.

The second post notes that people keep getting themselves into a bind, with some helpful tips on extricating yourself from those situations.

The fun thing to note here is the kinds of problems that we need to solve with git:

  • I missed a file from a commit
  • I put the wrong commit message
  • I put committed a change on the wrong branch
  • I need to remove a commit

To me, the fascinating thing about all of these things is that in the systems that came before git, it wasn’t possible to do any of them. Once you made a commit in Subversion, that was it. It was in the repository and immutable1.

Missed a file from a commit? Shrug

Committed something you shouldn’t have? Tough2

Also, in the late 2000s, when git was just starting to take off, an argument against switching was that the commit history was not sacrosanct; the record could be changed. Subversion’s behaviour was a feature not a bug.

In the next decade, we’ve changed mindset so much that the ability to change a commit is now expected, even when we know that it can cause problems.

Similarly, while git’s distributed nature certainly adds to its complexity, it’s both fantastically useful and not a source of its worst behaviours. I can sit on a plane with no internet access and still have most features. I can move my repo between cloud providers or host my own. When my broadband or AWS goes down, I can continue working.

Maybe rather than move to a proprietary, cloud-based source control system — which is what the author of the first piece is really advocating — we should remember what we gained when we moved last time.

Git is tricky. But rather than return to a centralised system maybe we should fix git’s user interface. Being able to fix past commits is useful, though we we should be able to do so without messing up our code or, especially, the code repository.

  1. As ever, it was always possible to hack the repo but in my experience this was frowned upon. ↩︎
  2. I do remember a time when someone committed a password. It took a lot of time and expertise to scrub it from the history. ↩︎

Why Enterprise Software is Bloated

I confess that I’ve stolen the title of the post from elsewhere. My objective here is not to detract from that post, which is great, but to build on a few points that I thought it was going to make but didn’t. To make it clear where I’m coming from, here’s a Tweet that I wrote some time before I read that blog:

People who complain that “enterprise” software is too expensive have clearly never come across the bizarre, arbitrary and nonsensical policies and rules these companies have. It’s not unusual to have two customers with contradictory policies.

Alex’s post assumes that bloat is unnecessary and could be optimised away with better engineering. There are always counter-examples, but, in general, this is not my experience.

The key is to realise two facts:

  1. Large companies have lots of rules and procedures
  2. Large companies have a lot of leverage over software companies

My favourite rules and processes are always about corporate security:

The game of cat and mouse begins: I need to send a file to a client, but which method will actually make it through their corporate security? So far I’m down 2–0 but I’m not giving up yet.

Corporate security is focused very much on stopping Bad Things from happening. Undoubtedly that’s one part of the job. The part that usually gets forgotten is to allow people to do their jobs securely. Without these “safe routes,” the security is making everyone’s life harder and increasing costs but without improving security.

Here’s a lightly fictionalised example. A client has a production outage. We need to see their log files to diagnose the problem. Dropbox, Box and Google Drive are all blocked. They don’t have a corporate file share. It’s not possible to email them, because the logs are large and ZIP files are forbidden. We can’t screen share, since they blocked Zoom (our tool) and corporate policy means they can’t open a Team’s call with outside people. There is a special dispensation for this, but it’s only available for client-facing staff. And, in any case, needs to be applied for in advance. In the end, we went Old Skool, zipped the files and renamed them with a “TXT” file extension and that worked.

There’s a lot to unpack here. First: the security measures didn’t work. Despite the obstacles, we managed to transfer the files. It just took multiple well paid people a lot of time to figure out. There was no known supported tool available to facilitate this use case which, let’s not forget, was an outage of one of their production systems, something that customers pay them actual money to be able to access.

As ever, the individual rules make some sense, but together they make a system that doesn’t work for anyone. It annoys staff, adds delays to tasks and in many cases does not work anyway.

Let’s circle back to the original question: why is enterprise software bloated?

The short answer is that vendors need to be able to support enterprises despite all that unique processes. A task, like copying log files, that should take minutes might take hours. Ultimately that needs to be paid for.

Many of the processes and procedures are unique to an individual enterprise. This means that the vendor has no option but to write custom software to accommodate them. These changes get folded back into the product and every other customer gets a copy of them, but hidden behind a feature flag.

Multiply this by every customer and you end up with a vast number of options, the majority of which are not used by most customers. It’s a bit crazy. It’s bloated, hard to keep track of and hard for the vendor to test.

If everyone recognises the problem, what is the solution?

Some companies just lay down the law: this is the right way to do something; take it or leave it. You can do this for consumer software or if you have millions of customers. There’s only so much negotiation you can do with Microsoft for your Windows or Office licence.

Companies that do this for “enterprise” software tend not to be very successful. There’s a reason that Google Cloud is a distant third, well behind AWS and Azure.

You can make “consultingware,” that is, software with a core that it heavily modified for each customer. At a certain point you don’t even have a product any more, with disadvantages for both the vendor and the user.

Or you can make a single product with switches that enable or disable all these esoteric options. Either way, all your customers end up paying extra for the privilege.

In the end, as long as big companies continue to customise their environments in mutually incompatible ways, “enterprise software” is going to be expensive and bloated.

Fragile Development

The problem with “agile development” is that it is both a methodology and a buzzword. What this means in practice is that people who do not understand it implement parts of it without appreciating the whole. This usually results in more overhead but without the benefits.

I’ve come across this multiple times in my career. The usual refrain is “we’re agile so we don’t need documentation.” The “agile” aspect is more often than not, merely the assertion that the project is agile. Or someone says that the code is the documentation.

Another common one is “we need to improve communication so we’ll have have a daily stand-up.” This often ends up being an opportunity for senior managers to give ten minute monologues about what they might get up to if they didn’t have so many meetings to attend. (Cue unheeded calls for sympathy.) Updates from people at the coal face often get cut due to time pressures. After a few weeks people stop attending…

Agile as it was originally designed was clever because it went up against the common wisdom of the day by jettisoning certain elements of bureaucracy but balancing them with other carefully considered, and hopefully less onerous, procedures. Less documentation might work if you increase the amount of teamwork; less formal requirements gathering is counteracted by including a user in the team; regular cadence of releases means you can get away with less rigid planning.

The common trait in the examples I gave is that they lack that balance. Not writing documentation is a mistake if there is no other method of retaining knowledge inside an organisation. Poor communication is not solved by managers broadcasting how out of touch they are.

As with any problem, you can’t fix it if you don’t understand it.

What do you know?

How do you interview people for developer and technical jobs? This is an enduring question, and one with many angry factions.

It’s too big a subject to tackle in its entirety and I have no intention of trying. Instead, I want to talk about one aspect: should you ask Computer Science questions or not?

In one corner are the people who argue that you never need to implement a linked list or write Quick Sort in real life, so asking you to do that in an interview is unreasonable and excludes good candidates. They argue that there are more important things to consider, such as the use of applications frameworks or design or working with other people.

In the other are those who say that algorithms are a fundamental part of writing software and, while you may not need to write a Quick Sort, you do need to understand it and to be able to explain it.

The first group calls the second elitist. The second calls the first naive. Who’s right?

Of course, I come at this with my own bias. I have a Computer Science degree but I did it some time ago and don’t have perfect recall on this stuff. Ask me the Big O notation for Quick Sort and I’ll understand the question and maybe make a stab at the answer, but I’m not going to pretend I’m 100% sure.

There are two main angles I want to consider: the intent of the interviewer; and the knowledge a developer actually needs to do the job.

Let’s start with the former.

So, your interviewer asks you to implement a linked list on a whiteboard. Why? What are they hoping to find out about you?

If the answer is… well, the answer, then I’m very firmly with the “you don’t need to know this stuff” crowd. People tend to want a specific answer when either they want a cookie cutter computer science graduate or they wouldn’t understand the answer.

Alternatively they might be trying to see how you think; how you’d start with a simple solution and build up; how you’d test it. In these cases, the answer is less important than how you got there.

Another thing to consider is that, as an interviewer, you want to get the most information in the least amount of time. Starting with a “real” business problem might require too much context to be explained before you could really start. But the candidate hopefully already know what a linked list does, even if they’ve never had to write one.

Of course, sat in a conference room with a stranger it’s very hard to tell which of the two camps your interviewer falls into (and practically impossible if they’re on the phone).

So that’s pretty inconclusive. There are good reasons to ask but you can’t tell whether that’s the case for your interviewer.

So how about job requirements?

This angle is easier. Most jobs don’t require standard algorithms that you’d come across in a computer science or software engineering degree. A knowledge of user interface frameworks probably is more useful for many projects.

But what about the rest? There are jobs where knowing algorithms is important. On projects where low-level or performance intensive code is required understanding the fundamentals can be important. Maybe you’ll be writing a game or something with very low latency requirements or an engine that processes vast amounts of data.

In my last project I deliberately avoided using the system provided Quick Sort and implemented my own Heap Sort. I may not have been able to tell you the Big O notation for heap sort and quick sort, but I do know that quick sorts’ Achilles heel is that it works poorly on already sorted data. (It’s only fair to note that they didn’t ask me about algorithms when I interviewed for that position. In fact, they didn’t ask me much of anything! That’s a long story for another time.)

I’m not saying that you need a Computer Science degree to know those things. I’m not saying that the project would have failed without that background (though the fact that someone knew saved tens of thousands of pounds in hardware costs). And, most importantly, I’m not saying that the kinds of projects that you do are anything like mine. I am saying that the people who say this stuff isn’t useful or important are wrong.

So what do we conclude?

Well, firstly, and most importantly, no one size fits all. Your interview process needs to be tuned for the kinds of position you’re trying to fill.

Secondly, if the only questions are about linked lists and sort algorithms, that’s a big red flag. These aren’t the only interesting subjects for any job. If you’re not at least asked about how you work in teams, you should be worried.

Finally, having said all that, you can’t escape from the fact that programming is about algorithms. According to Wikipedia:

Programming involves activities such as analysis, developing understanding, generating algorithms

You should be expected to understand and to be able to explain algorithms.

But you should also understand team work, several programming languages, relevant frameworks, user interface design, source control, your IDE, the Unix command line, security, Linux, web servers, leadership and management, the film that the line “pop quiz, hotshot” comes from and know the correct answers to “tabs or spaces?” Developing software is hard and has many facets.

Focusing on any one aspect to the exclusion of others is a mistake. And that’s true both for you as an individual and you as an employer.

Double Trouble

One of the great things about WordPress is the community and the number of great plugins that can do amazing things with little effort.

But all that code, as any good developer will tell you, is a liability. How do you pick a plugin that not only meets your requirements now, but will both continue to do so? WordPress advances. APIs change. Plugins need love too.

Many moons ago I settled upon Flickr Gallery, a plugin that allows you to import Flickr images just by adding a short-code to posts. I thought there was value in keeping all my public photos in one place and, at that time, WordPress had poor media management facilities. The plugin seemed popular and well supported.

Flash forward to 2015 and, well, it doesn’t work. It hasn’t been updated for nearly five years and where I should see pictures I only see blanks.

But I’m a developer. How hard can it be to fix?

Groan! Have I ever said how much I hate web development?

Anyway… two hours later I find that there are not one but two problems. I find this after fixing the first problem in a server running on my local machine but find that it still doesn’t work here.

The first problem is that Flickr now requires SSL access to its API. In code terms, open phpFlickr.php and change the following lines:

 var $rest_endpoint = 'http://api.flickr.com/services/rest/';
 var $upload_endpoint = 'http://api.flickr.com/services/upload/';
 var $replace_endpoint = 'http://api.flickr.com/services/replace/';


 var $rest_endpoint = 'https://api.flickr.com/services/rest/';
 var $upload_endpoint = 'https://api.flickr.com/services/upload/';
 var $replace_endpoint = 'https://api.flickr.com/services/replace/';

I found the cause of the second problem when I realised that none of the flickr code was getting called. It turns out that I have the Jetpack plugin and that also uses the “flickr” shortcode, though the syntax for using it is slightly different.

(I find it odd that there’s no error or warning flagging the conflict. This took far longer to track down than it needed to.)

I tried disabling that option in the Jetpack plugin but that didn’t work. In the end, I added the following code to flickr-gallery.php, just before the “add_shortcode(‘flickr’…” line:

if (shortcode_exists('flickr')) {

Very much cheating… but it does work.

I’m trying to figure out if there’s a way of distributing the change in a more user-friendly format.