Tag Archives: Software

History

A few years ago I had a job where every new recruit would go through a long process of shock and gradual acclimatisation to the main software product.

What it did doesn’t matter as much as how it was built: it was an application developed on top of a proprietary programming language and user interface designer. The reaction was always the same. Why? Why?! Why would you reinvent Visual Basic on Unix? Why would you inflict a programming language even worse than Basic on developers?1

The answer, it turns out, is that the original developers were idiots.

No, of course that’s not true. But if that’s the case, then why did almost every developer start from that point of view when they first arrived at the company?

That brings us to Twitter and its new owner. One of his first public proclamations is to declare that there are too many micro-services running, and, worse, most of do nothing useful! The reply-guys all agree and, between them, argue that it’s entirely possible to rebuild Twitter from the ground-up in weeks, possibly even a weekend if given enough pizza and Blue Bottle.

Were the original developers of Twitter also idiots?

I don’t know as much about Twitter’s architecture, but I’d be willing to bet that, no, they were also not stupid.

If it’s not the original developers, what does it say about the critic? It says that they see the complexity but not the nuance. They see the current state but they do not see any of the decisions that lead up the current system. They see complexity, but without understanding the whole problem domain they don’t see why that complexity exists.

In the case of my job, the software predated Visual Basic, which is a pretty good reason for not using it. It also had to work on Unix and be editable on client sites without extra tooling. By the time I worked there, it may have been dated but it was in production at many clients. It worked. Sure, it’s not how you’d architect it now but the decisions that led to the design did make sense.

If it’s dated, then why not rewrite it? That has been covered many times before, but the short answer is that when you design it, you focus so much on the clean, new solution that you forget why you added the warts to the old system. The layers upon layers of fixes and enhancements represent real world experience. Those micro-services are there for a reason. Not understanding the reason doesn’t change that2.

This is not an argument against evolving the software, only that you should understand what you already have. Sometimes rewriting can be justified. Rationalising a bunch of micro-services isn’t always a ridiculous idea. But there’s an important difference between complex and complicated. Can you know which your inherited system is after a few days on the job?


  1. It was a stack-based language, along the lines of Forth and Postscript. Long time users could do amazing things with tiny amounts of code. I never quite got there. ↩︎

  2. Logical fallacy: argument from incredulity. ↩︎

Facts and Fallacies of Software Engineering

I’ll be honest: I wanted to like “Facts and Fallacies of Software Engineering” by Robert L. Glass more than I did. I’m not sure if it’s dated badly — it’s from 2002 — or I was in the wrong frame of mind, or something else, but it just didn’t work for me.

The book is structured as a list of facts grouped around areas such as “Management” and “Requirements.” For each fact, there is a discussion, the controversy, and then the sources and references. The writing aims to be friendly, but I found it a bit grating1.

Going back through the fifty-five facts and fallacies, I find that I agree with the majority of them2. Twenty years on from the original version, I suspect that some are less controversial than when it was originally published. There is some degree of (justified) scepticism about the newfangled “agile” process, which is now largely standard across the industry. Yet it’s also reassuring how little other things have changed. There is still little transfer between academia and industry. Quality continues to be a hot-button topic. And software maintenance rarely gets the love it deserves.

I’ve seen this book described as a classic, so maybe I’m missing something. There are many books I would suggest you read first. In the end, I didn’t like the writing, and I think that there are too few original insights in too many words.


  1. This is such a subjective thing. I think the objective in was modesty or self-deprecation, but it came across as boasting. ↩︎
  2. I’m resisting the temptation to argue against those that I don’t agree with. ↩︎

Why Enterprise Software is Bloated

I confess that I’ve stolen the title of the post from elsewhere. My objective here is not to detract from that post, which is great, but to build on a few points that I thought it was going to make but didn’t. To make it clear where I’m coming from, here’s a Tweet that I wrote some time before I read that blog:

People who complain that “enterprise” software is too expensive have clearly never come across the bizarre, arbitrary and nonsensical policies and rules these companies have. It’s not unusual to have two customers with contradictory policies.

Alex’s post assumes that bloat is unnecessary and could be optimised away with better engineering. There are always counter-examples, but, in general, this is not my experience.

The key is to realise two facts:

  1. Large companies have lots of rules and procedures
  2. Large companies have a lot of leverage over software companies

My favourite rules and processes are always about corporate security:

The game of cat and mouse begins: I need to send a file to a client, but which method will actually make it through their corporate security? So far I’m down 2–0 but I’m not giving up yet.

Corporate security is focused very much on stopping Bad Things from happening. Undoubtedly that’s one part of the job. The part that usually gets forgotten is to allow people to do their jobs securely. Without these “safe routes,” the security is making everyone’s life harder and increasing costs but without improving security.

Here’s a lightly fictionalised example. A client has a production outage. We need to see their log files to diagnose the problem. Dropbox, Box and Google Drive are all blocked. They don’t have a corporate file share. It’s not possible to email them, because the logs are large and ZIP files are forbidden. We can’t screen share, since they blocked Zoom (our tool) and corporate policy means they can’t open a Team’s call with outside people. There is a special dispensation for this, but it’s only available for client-facing staff. And, in any case, needs to be applied for in advance. In the end, we went Old Skool, zipped the files and renamed them with a “TXT” file extension and that worked.

There’s a lot to unpack here. First: the security measures didn’t work. Despite the obstacles, we managed to transfer the files. It just took multiple well paid people a lot of time to figure out. There was no known supported tool available to facilitate this use case which, let’s not forget, was an outage of one of their production systems, something that customers pay them actual money to be able to access.

As ever, the individual rules make some sense, but together they make a system that doesn’t work for anyone. It annoys staff, adds delays to tasks and in many cases does not work anyway.

Let’s circle back to the original question: why is enterprise software bloated?

The short answer is that vendors need to be able to support enterprises despite all that unique processes. A task, like copying log files, that should take minutes might take hours. Ultimately that needs to be paid for.

Many of the processes and procedures are unique to an individual enterprise. This means that the vendor has no option but to write custom software to accommodate them. These changes get folded back into the product and every other customer gets a copy of them, but hidden behind a feature flag.

Multiply this by every customer and you end up with a vast number of options, the majority of which are not used by most customers. It’s a bit crazy. It’s bloated, hard to keep track of and hard for the vendor to test.

If everyone recognises the problem, what is the solution?

Some companies just lay down the law: this is the right way to do something; take it or leave it. You can do this for consumer software or if you have millions of customers. There’s only so much negotiation you can do with Microsoft for your Windows or Office licence.

Companies that do this for “enterprise” software tend not to be very successful. There’s a reason that Google Cloud is a distant third, well behind AWS and Azure.

You can make “consultingware,” that is, software with a core that it heavily modified for each customer. At a certain point you don’t even have a product any more, with disadvantages for both the vendor and the user.

Or you can make a single product with switches that enable or disable all these esoteric options. Either way, all your customers end up paying extra for the privilege.

In the end, as long as big companies continue to customise their environments in mutually incompatible ways, “enterprise software” is going to be expensive and bloated.

Generalist Software Engineering

I greatly enjoyed Graham Lee’s series of posts about specialisation versus generalisation in software engineering1, quite possibly because it’s me.

My background is a little different from Lee’s, though, so I thought it was worth sharing.

I have a two tier experience2. With a few minor blips, Unix has been a constant technology underpinning since my first year at university. I started using Linux around the time 1.0 was released. I got a Mac when — or possibly before — OS X was ready for mainstream use because it was Unix with a nice UI. At work I’ve seen the change from big Solaris and HP-UX machines, to Linux, to containerised applications (which are normally based on minimal Linux distributions). Sure, the different Unix variants are not exactly the same, but most of them have something bash-like and ls does the same thing everywhere, even if the more esoteric options vary.

I should say that this is largely a preference. I don’t like Windows but that’s not an objective criticism. I do joke about it from time to time3 and I do admit the limits on my knowledge, but I don’t refuse to work with it!

Sadly, being a long-time Unix user is not a career.

I started my career at a “pure” consulting company. Each client I worked with wanted to do something different and I ended up using varied technology stacks. This was great from a “generalist” point of view. I flitted from Uniface to PL/SQL to Perl to Oracle Applications4, but beyond abstract concepts like “problem solving” I didn’t get anything like a transferable skill. The obvious path would have been management but that wasn’t where I wanted to go. I saw friends and colleagues specialising, and earning considerably more money for the privilege.

I appreciate that I’m likely leaving money on the table, but I stumbled on a solution that works for me: being part of the field engineering team of software vendors.

Being in the field team means that I work directly with customers and they end up wanting to do all kinds of things. In a year I can work with dozens of use cases, satisfying my need for novelty. At an end-user, I’d be looking at a small number of use cases the whole time.

At the same time, there is also a degree of specialisation. By working with the same product all the time, I can become The Expert in it and some adjacent technologies. For example, I’m pretty good with Kubernetes and Java these days. I need to write code and sketch out software architectures. My knowledge has to be deep enough to demonstrate credibility but I don’t have to build production code. Additionally, I can bring domain expertise. I wouldn’t sell myself purely on my banking knowledge, but it’s a nice value add.

This may not be the solution for you. There may even be better options for me that I’ve not found yet! But I thought it was worth documenting my experience, since most of the articles I’ve seen are for more traditional “software engineering” or management roles. Other positions are out there if you know where to look.


  1. Though I wasn’t able to write anything about it in a timely manner! ↩︎
  2. Maybe second tier too, but in this case I mean there are at least two layers. ↩︎
  3. There’s a Dilbert cartoon I use occasionally. This from before Adams went off the rails. ↩︎
  4. Whether you’d want to make a career out of any of those technologies is a different story. ↩︎

Ops is undervalued

I made the mistake of suggesting that there was a blog to be made from this tweet. This is that post.

People still underestimate the value in (Developer|Operator) Experience when building platforms and honestly it’s kinda shocking to me.

If you want to win mindshare you need to make your tools actually usable. If you don’t want to lose it you need the same.

First: I agree with the sentiment. Maybe not to the same extent as Danielle, but I fight the same battles in my day job. I wanted to say this now because, on reading the rest, you may think the opposite. What follows is an explanation of why this is a common situation. I don’t mean it as a justification.

In summary, making software work for ops teams is not a focus either for software companies or internal development teams because of at least three reasons:

  • It’s not a business driver
  • Ops are not the buyer
  • Engineering is run by developers

Naturally there are exceptions to these rules. Every company is different. These are observations, not rules.

Working at the “coal face” it’s easy to forget, but people don’t buy software. They buy a solution to a business problem1. These days, that solution is often software but you don’t buy a new product because it’s cool2.

A driver might be to get a report generated more quickly, or to provide a new service to paying customers, or to reduce the costs of some infrastructure3.

But, you argue, the ops team are the people who keep the system up and running. How can this new system generate reports or deliver a service if it’s not running reliably or has not been provisioned correctly?

You’d be right. Sadly, organisations are often not structured to recognise that fact. The IT teams tasked with making everything run smoothly are frequently in a completely different reporting hierarchy from “the business.” I put “the business” in quotes as I hear it described that way all the time, but it’s this us-versus-them philosophy that brings many issues, including how ops get sidelined4.

With “the business” and “ops” being in separate reporting structures, one or the other has to sign off on the purchase of new software (unless it goes way up the organisation) and that’s always going to be “the business.”

The buyers normally consult the ops team, but ultimately they’re going to put their own needs first. Objections that the ops team have will end up in the business case, but likely in an appendix that no one will ever read.

This makes no sense because, as we all know, most of the expense of a system occurs after go-live. But monitoring, management and deployment are not things at the top of mind of developers and business users.

But even in the IT organisation, the ops team frequently don’t get the attention they deserve either. Anecdotally, this is because IT leadership come from the development team. Their outlook on IT is therefore skewed towards making things rather than keeping them running. I see the same challenges for testing teams. Or, indeed, the lack of testing teams.

This lack of buy-in from the ops team is endemic. I see it in companies buying and using software. I see it in companies that make and sell software5.

At this point, what I would like to be able to do is say, “And the solution to this terrible problem is…” Sadly there is no easy answer. Saying “You should listen to your ops team” is both obvious and unhelpful. Making tools useful for the ops team to get mindshare is a good way to get your software on a shortlist but maybe not enough to clinch that sale.


  1. I’m talking here about software used in a business of course, but the difference between this and personal use isn’t as great as you might initially imagine. You buy a game to solve the “problem” of boredom. Very few people buy software because it’s software. ↩︎
  2. You might buy it because this, specific solution is cooler than the other options. ↩︎
  3. That is, even in the case where it’s the infrastructure that is being improved, the business case is not “make the ops team more efficient” but “make the cost of operations lower.” ↩︎
  4. There’s another blog in how toxic “the business” as a concept is. This isn’t that blog. ↩︎
  5. This is a chicken and egg problem. Do software companies fall into this trap because they’re run by developers? Or is it because they’re selling to companies who have already fallen into the trap? ↩︎

CameraGPS debrief

As happy as I am with the way that my new app, CameraGPS, a GPS logger application for people who want to geotag their photographs, came out I can’t say that it’s exactly as I envisioned it at the start of the process.

The idea was something like this: many of the GPS logger apps in the App Store require you to either use iTunes file sharing (who connects their iPhone’s to iTunes any more?) or mail yourself the exported document or sign up to some third party fitness or trekking website. Mailing yourself stuff just didn’t feel very slick and I didn’t want to record my trails for fitness purposes.

I felt that there must be a better way. The better way, I thought, was to use iCloud to share the files between iOS and a Mac. This would require writing a recorder app for iPhone and an “exporter” app for the Mac.

It didn’t work out that way in the end. Here’s why.

You see all kinds of horror stories from people who have had to deal with iCloud. Most of those stories come from people who used the Core Data syncing, while there are a few apps that successfully use the document syncing model, not least some of Apple’s own. Using the document storage I figured that, worst case, if something gets screwed up it would only affect a single trail. Using Core Data syncing, an error could mean corrupting the whole database.

So this became the plan: record each GPS trail into a separate document. Because I wanted to “random access” parts of each trail and because I didn’t want to have to load the whole trail into memory all the time, I decided to use Core Data to manage the document. On iOS this is called UIManagedDocument.

The first and most obvious fail here is with UIManagedDocument’s cross platform capabilities. Or rather the lack of them. Unless you count iPhone to iPad as cross platform it just doesn’t work. There is a class on the Mac called NSPersistentDocument that is also a Core Data store and looks on the surface to do exactly the same thing… but it’s in a different format. On iOS it’s a “package” which includes a Core Data store. On the Mac it’s just the store. Mike Abdullah has written an open source component that attempts to be bridge the gap but — as we’ll see shorty — I didn’t get to the point where I was able to test it in anger.

But let’s take a step backwards. Let’s assume that all we need is to move data between iPhone and iPad. Then what?

Well, technically it works — in the sense that documents transfer between machines — just not in a way that would make sense to ship.

To make sense of why it doesn’t work let’s take a quick diversion to look at the format of a UIManagedDocument. As hinted above, it’s not a file. Rather it’s a “package” which a number of files within.

An important file is called DocumentMetadata.plist. This tells you about the rest of the package and is created when the document is. There’s little in here that needs to be changed after document creation. The interesting stuff is all stored in a SQLite file, which is stored in a folder.

When files are stored in iCloud you can’t just use NSFileManager to poke around and see what’s there. Instead you create a query using NSMetadataQuery — which supports both a predicate and a sort descriptor — and listen for updates. Updates come in two parts, the initial state and then changes to that state.

After a few false starts I ended up using KVO to populate a table view showing the current list of documents. This worked nicely, showing the full list, updating automatically when changes arrived over the network.

But, and it’s a big but, what if you want to sort the list. Pretty basic requirement, no?

One of the fun things about NSMetadataQuery is that it can only look for files. Folders are “invisible.”

Most of Apple’s documentation — where there is any — suggests searching for the DocumentMetadata.plist. Which makes sense but with two significant drawbacks: every file that you’re looking for now has exactly the same name; and the modification date is the creation date of the document rather than the time of the last update.

This means that you can’t sort the documents alphabetically or by modification date using only the metadata predicate. To get the names you need to listen for updates, copy them elsewhere and then butcher the file name, that is removing the last component and possibly the file extension.

To get the last modification date you probably need to open the document and do a Core Data query. This sounds pretty straight-forward but, again, there are difficulties. Remember that NSMetadataQuery tells us what’s available in iCloud rather than what has been downloaded onto the current device. Just because the query tells us the file is in iCloud doesn’t mean that you can open it immediately.

UIDocument helps us a bit here: if you ask to open a document it will download it if necessary. However there’s a big difference between opening a local file and downloading one over the network and then opening it. Also remember, at this point we’re not opening it as the result of a user request. We’re just trying to find the last modification date so that we can sort the list of documents conveniently.

None of this, of course, is impossible. It’s just a lot of faff for something that pretty much every app that wants to use UIDocument must have to go through.

Okay, so we get all that done. We’re good to go now, right? Not quite.

Part of the idea of using a UIDocument was that it would appear as a, well, document. But that whole thing about what NSMetatadataQuery “sees” comes back to haunt us.

The short, weird version is that if you configure the application to look for DocumentMetadata.plist, then that’s what is seen in the Settings app. Not very helpful as all the documents will all have the same name.

On the other hand you can configure the app to “know” about the document type. This means that the Settings app understands that they’re document and displays them with the correct “file” name. Unfortunately then the app can’t see the documents properly. Specifically, you can search for the document by name — which is nice — but you can’t actually open it. It’s odd and I’m not entirely sure what goes wrong, but when you call openWithCompletionHandler: it never actually calls the completion handler and the document never becomes available for use. (I initially thought that I just wasn’t being patient enough. I later accidentally left the app running for over two hours and the completion handler still never ran. That convinced me that it wasn’t a patience thing.)

I convinced myself that this couldn’t possibly be how it was supposed to work. My question on Stack Overflow didn’t get any answers and a reply to another, similar question said that the behaviour was “by design.”

I think it’s just broken and that no one is really using it. Either that or there’s a secret handshake that you have to give to the right person at WWDC.

So what’s shipping? Pretty much none of the above. I read about the new iOS 7 methods of Core Data – iCloud syncing and liked what I saw. Additionally iOS 7 comes with a few more APIs with better error handling hooks. This made me happier shipping with Core Data syncing. Just using “raw” Core Data meant that I could simplify a lot of code, use NSFetchedResultController and remove a bunch of the more fragile iCloud code (mostly NSMetadataQuery). While I was quite pleased with the KVO code I wrote, having far less code overall is a much bigger win.

And, in case you were wondering, I also de-scoped the Mac app and integrated Dropbox support instead.

But the bottom line is that this is all a lot harder than it should be. My concerns originally were around error conditions — something that has been very weak in iCloud since it’s introduction in iOS 5 — but it quickly morphed into questions about how it’s all supposed to work at all.

I once quipped on Twitter that you can tell which APIs are the ones Apple uses themselves as they’re the well designed ones. I think it’s pretty clear that Apple don’t use UIManagedDocument.