Tag Archives: development

Fragile Development

The problem with “agile development” is that it is both a methodology and a buzzword. What this means in practice is that people who do not understand it implement parts of it without appreciating the whole. This usually results in more overhead but without the benefits.

I’ve come across this multiple times in my career. The usual refrain is “we’re agile so we don’t need documentation.” The “agile” aspect is more often than not, merely the assertion that the project is agile. Or someone says that the code is the documentation.

Another common one is “we need to improve communication so we’ll have have a daily stand-up.” This often ends up being an opportunity for senior managers to give ten minute monologues about what they might get up to if they didn’t have so many meetings to attend. (Cue unheeded calls for sympathy.) Updates from people at the coal face often get cut due to time pressures. After a few weeks people stop attending…

Agile as it was originally designed was clever because it went up against the common wisdom of the day by jettisoning certain elements of bureaucracy but balancing them with other carefully considered, and hopefully less onerous, procedures. Less documentation might work if you increase the amount of teamwork; less formal requirements gathering is counteracted by including a user in the team; regular cadence of releases means you can get away with less rigid planning.

The common trait in the examples I gave is that they lack that balance. Not writing documentation is a mistake if there is no other method of retaining knowledge inside an organisation. Poor communication is not solved by managers broadcasting how out of touch they are.

As with any problem, you can’t fix it if you don’t understand it.

What do you know?

How do you interview people for developer and technical jobs? This is an enduring question, and one with many angry factions.

It’s too big a subject to tackle in its entirety and I have no intention of trying. Instead, I want to talk about one aspect: should you ask Computer Science questions or not?

In one corner are the people who argue that you never need to implement a linked list or write Quick Sort in real life, so asking you to do that in an interview is unreasonable and excludes good candidates. They argue that there are more important things to consider, such as the use of applications frameworks or design or working with other people.

In the other are those who say that algorithms are a fundamental part of writing software and, while you may not need to write a Quick Sort, you do need to understand it and to be able to explain it.

The first group calls the second elitist. The second calls the first naive. Who’s right?

Of course, I come at this with my own bias. I have a Computer Science degree but I did it some time ago and don’t have perfect recall on this stuff. Ask me the Big O notation for Quick Sort and I’ll understand the question and maybe make a stab at the answer, but I’m not going to pretend I’m 100% sure.

There are two main angles I want to consider: the intent of the interviewer; and the knowledge a developer actually needs to do the job.

Let’s start with the former.

So, your interviewer asks you to implement a linked list on a whiteboard. Why? What are they hoping to find out about you?

If the answer is… well, the answer, then I’m very firmly with the “you don’t need to know this stuff” crowd. People tend to want a specific answer when either they want a cookie cutter computer science graduate or they wouldn’t understand the answer.

Alternatively they might be trying to see how you think; how you’d start with a simple solution and build up; how you’d test it. In these cases, the answer is less important than how you got there.

Another thing to consider is that, as an interviewer, you want to get the most information in the least amount of time. Starting with a “real” business problem might require too much context to be explained before you could really start. But the candidate hopefully already know what a linked list does, even if they’ve never had to write one.

Of course, sat in a conference room with a stranger it’s very hard to tell which of the two camps your interviewer falls into (and practically impossible if they’re on the phone).

So that’s pretty inconclusive. There are good reasons to ask but you can’t tell whether that’s the case for your interviewer.

So how about job requirements?

This angle is easier. Most jobs don’t require standard algorithms that you’d come across in a computer science or software engineering degree. A knowledge of user interface frameworks probably is more useful for many projects.

But what about the rest? There are jobs where knowing algorithms is important. On projects where low-level or performance intensive code is required understanding the fundamentals can be important. Maybe you’ll be writing a game or something with very low latency requirements or an engine that processes vast amounts of data.

In my last project I deliberately avoided using the system provided Quick Sort and implemented my own Heap Sort. I may not have been able to tell you the Big O notation for heap sort and quick sort, but I do know that quick sorts’ Achilles heel is that it works poorly on already sorted data. (It’s only fair to note that they didn’t ask me about algorithms when I interviewed for that position. In fact, they didn’t ask me much of anything! That’s a long story for another time.)

I’m not saying that you need a Computer Science degree to know those things. I’m not saying that the project would have failed without that background (though the fact that someone knew saved tens of thousands of pounds in hardware costs). And, most importantly, I’m not saying that the kinds of projects that you do are anything like mine. I am saying that the people who say this stuff isn’t useful or important are wrong.

So what do we conclude?

Well, firstly, and most importantly, no one size fits all. Your interview process needs to be tuned for the kinds of position you’re trying to fill.

Secondly, if the only questions are about linked lists and sort algorithms, that’s a big red flag. These aren’t the only interesting subjects for any job. If you’re not at least asked about how you work in teams, you should be worried.

Finally, having said all that, you can’t escape from the fact that programming is about algorithms. According to Wikipedia:

Programming involves activities such as analysis, developing understanding, generating algorithms

You should be expected to understand and to be able to explain algorithms.

But you should also understand team work, several programming languages, relevant frameworks, user interface design, source control, your IDE, the Unix command line, security, Linux, web servers, leadership and management, the film that the line “pop quiz, hotshot” comes from and know the correct answers to “tabs or spaces?” Developing software is hard and has many facets.

Focusing on any one aspect to the exclusion of others is a mistake. And that’s true both for you as an individual and you as an employer.

Double Trouble

One of the great things about WordPress is the community and the number of great plugins that can do amazing things with little effort.

But all that code, as any good developer will tell you, is a liability. How do you pick a plugin that not only meets your requirements now, but will both continue to do so? WordPress advances. APIs change. Plugins need love too.

Many moons ago I settled upon Flickr Gallery, a plugin that allows you to import Flickr images just by adding a short-code to posts. I thought there was value in keeping all my public photos in one place and, at that time, WordPress had poor media management facilities. The plugin seemed popular and well supported.

Flash forward to 2015 and, well, it doesn’t work. It hasn’t been updated for nearly five years and where I should see pictures I only see blanks.

But I’m a developer. How hard can it be to fix?

Groan! Have I ever said how much I hate web development?

Anyway… two hours later I find that there are not one but two problems. I find this after fixing the first problem in a server running on my local machine but find that it still doesn’t work here.

The first problem is that Flickr now requires SSL access to its API. In code terms, open phpFlickr.php and change the following lines:

 var $rest_endpoint = 'http://api.flickr.com/services/rest/';
 var $upload_endpoint = 'http://api.flickr.com/services/upload/';
 var $replace_endpoint = 'http://api.flickr.com/services/replace/';

To:

 var $rest_endpoint = 'https://api.flickr.com/services/rest/';
 var $upload_endpoint = 'https://api.flickr.com/services/upload/';
 var $replace_endpoint = 'https://api.flickr.com/services/replace/';

I found the cause of the second problem when I realised that none of the flickr code was getting called. It turns out that I have the Jetpack plugin and that also uses the “flickr” shortcode, though the syntax for using it is slightly different.

(I find it odd that there’s no error or warning flagging the conflict. This took far longer to track down than it needed to.)

I tried disabling that option in the Jetpack plugin but that didn’t work. In the end, I added the following code to flickr-gallery.php, just before the “add_shortcode(‘flickr’…” line:

if (shortcode_exists('flickr')) {
    remove_shortcode('flickr');
}

Very much cheating… but it does work.

I’m trying to figure out if there’s a way of distributing the change in a more user-friendly format.

Swift Hate

I’m seeing a surprising amount of vitriol aimed at Swift, Apple’s new programming language for iOS and Mac development. I understand that there can be reasoned debate around the features (or lack thereof), syntax and even the necessity of it but there can be little doubt about the outcome: if you want to consider yourself an iOS developer, it’s a language that you will need to learn.

The only variable I can think of is when you learn it.

I think it’s perfectly reasonable to delay learning it as you have code to deliver now and because Swift is, well, very beta currently.

But asking what the point of Swift is not constructive. Asking what problems can be solved exclusively by Swift makes no sense at all you can do almost anything in most programming languages. Just because Intercal is Turing complete doesn’t mean that you’d want to use it for any real work. What varies between languages is what’s easy and what’s hard.

Objective-C undoubtedly makes some things easier than Swift. It’s a more dynamic language and its C foundations open up a lot of low-level optimisations that probably are not there in higher level languages.

But that same flexibility comes with a price: segmentation faults and memory leaks; pointers; easy-to-get-wrong switch statement; a lack of bounds checking. It also inherits a lot of ambiguity from original C language specification which makes certain automatic optimisations impossible.

How many applications require low-level optimisations more than safety? (And that’s ignoring that the biggest optimisations are usually found in designing a better algorithm or architecture.) How often is it better to have five lines of code instead of one? Every line is a liability, something that can go wrong, something that needs to be understood, tested and maintained.

Whatever its failings, it’s already clear that Swift is more concise and safer than Objective C.

There are absolutely applications where Objective C, C or C++ would be a better choice than Swift. It’s the old 80-20 rule applied to a programming language. And, for those resistant to learning a new language, the 80% is not on “your” side.

Right now, some of this requires a bit of a leap of faith. Swift clearly isn’t finished. You can either learn it now and potentially have some say on what the “final” version looks like, or you can learn it afterwards and just have to accept what’s there. But, either way, you’ll probably using it next year. Get used to it.

What you forgot from your Computer Science Degree

Last night I did a short presentation about my WSLHTMLEntities open source project at the London iOS Developer Group meeting. You can see the slides here:

Since last time I did a talk there people snickered because I built the slides using PowerPoint, this time I decided to use the latest Apple technology: Keynote in iCloud. Unfortunately this was a bit too new for the Mac Pro they use in the Apple Store, so we ended up downloading a copy in PowerPoint format and loading that into the local copy of Keynote. Nothing is ever simple.

One question I got at the end that I was unable to answer is how well it performs compared to other solutions.

This afternoon I ran a few quick tests.

Firstly I added two further methods of performing the HTML entity replacement:

The last option, while built-in and easy to implement, is not something you’d want to consider if you had any performance requirements. First, it has to run on the main thread. Second, it’s memory footprint is way higher than either of the other two solutions. And, finally, it’s slow. In my tests I did a loop of 10,000 iterations. Because of the memory problem I only ran 1000 iterations of the NSAttributedString solution, but it was still about 15 times slower than my version when doing the full 10,000 records (i.e., over 150 times slower overall).

So I’ve not included it in the charts below.

HTMLEntityChart

I ran eight simple tests with each of the parsers.

There is no clear winner.

As I speculated, WSLHTMLEntities is much more consistent than the other two. Google’s varies a lot depending on where in the mapping list the “solution” lies. Replacing ◊ (near the end of the list) takes 50% longer than & (near the beginning) for example.

Still, you’d have to be pretty obsessive to pick one of these solutions for performance reasons alone. Interesting to find out, though!

CameraGPS debrief

As happy as I am with the way that my new app, CameraGPS, a GPS logger application for people who want to geotag their photographs, came out I can’t say that it’s exactly as I envisioned it at the start of the process.

The idea was something like this: many of the GPS logger apps in the App Store require you to either use iTunes file sharing (who connects their iPhone’s to iTunes any more?) or mail yourself the exported document or sign up to some third party fitness or trekking website. Mailing yourself stuff just didn’t feel very slick and I didn’t want to record my trails for fitness purposes.

I felt that there must be a better way. The better way, I thought, was to use iCloud to share the files between iOS and a Mac. This would require writing a recorder app for iPhone and an “exporter” app for the Mac.

It didn’t work out that way in the end. Here’s why.

You see all kinds of horror stories from people who have had to deal with iCloud. Most of those stories come from people who used the Core Data syncing, while there are a few apps that successfully use the document syncing model, not least some of Apple’s own. Using the document storage I figured that, worst case, if something gets screwed up it would only affect a single trail. Using Core Data syncing, an error could mean corrupting the whole database.

So this became the plan: record each GPS trail into a separate document. Because I wanted to “random access” parts of each trail and because I didn’t want to have to load the whole trail into memory all the time, I decided to use Core Data to manage the document. On iOS this is called UIManagedDocument.

The first and most obvious fail here is with UIManagedDocument’s cross platform capabilities. Or rather the lack of them. Unless you count iPhone to iPad as cross platform it just doesn’t work. There is a class on the Mac called NSPersistentDocument that is also a Core Data store and looks on the surface to do exactly the same thing… but it’s in a different format. On iOS it’s a “package” which includes a Core Data store. On the Mac it’s just the store. Mike Abdullah has written an open source component that attempts to be bridge the gap but — as we’ll see shorty — I didn’t get to the point where I was able to test it in anger.

But let’s take a step backwards. Let’s assume that all we need is to move data between iPhone and iPad. Then what?

Well, technically it works — in the sense that documents transfer between machines — just not in a way that would make sense to ship.

To make sense of why it doesn’t work let’s take a quick diversion to look at the format of a UIManagedDocument. As hinted above, it’s not a file. Rather it’s a “package” which a number of files within.

An important file is called DocumentMetadata.plist. This tells you about the rest of the package and is created when the document is. There’s little in here that needs to be changed after document creation. The interesting stuff is all stored in a SQLite file, which is stored in a folder.

When files are stored in iCloud you can’t just use NSFileManager to poke around and see what’s there. Instead you create a query using NSMetadataQuery — which supports both a predicate and a sort descriptor — and listen for updates. Updates come in two parts, the initial state and then changes to that state.

After a few false starts I ended up using KVO to populate a table view showing the current list of documents. This worked nicely, showing the full list, updating automatically when changes arrived over the network.

But, and it’s a big but, what if you want to sort the list. Pretty basic requirement, no?

One of the fun things about NSMetadataQuery is that it can only look for files. Folders are “invisible.”

Most of Apple’s documentation — where there is any — suggests searching for the DocumentMetadata.plist. Which makes sense but with two significant drawbacks: every file that you’re looking for now has exactly the same name; and the modification date is the creation date of the document rather than the time of the last update.

This means that you can’t sort the documents alphabetically or by modification date using only the metadata predicate. To get the names you need to listen for updates, copy them elsewhere and then butcher the file name, that is removing the last component and possibly the file extension.

To get the last modification date you probably need to open the document and do a Core Data query. This sounds pretty straight-forward but, again, there are difficulties. Remember that NSMetadataQuery tells us what’s available in iCloud rather than what has been downloaded onto the current device. Just because the query tells us the file is in iCloud doesn’t mean that you can open it immediately.

UIDocument helps us a bit here: if you ask to open a document it will download it if necessary. However there’s a big difference between opening a local file and downloading one over the network and then opening it. Also remember, at this point we’re not opening it as the result of a user request. We’re just trying to find the last modification date so that we can sort the list of documents conveniently.

None of this, of course, is impossible. It’s just a lot of faff for something that pretty much every app that wants to use UIDocument must have to go through.

Okay, so we get all that done. We’re good to go now, right? Not quite.

Part of the idea of using a UIDocument was that it would appear as a, well, document. But that whole thing about what NSMetatadataQuery “sees” comes back to haunt us.

The short, weird version is that if you configure the application to look for DocumentMetadata.plist, then that’s what is seen in the Settings app. Not very helpful as all the documents will all have the same name.

On the other hand you can configure the app to “know” about the document type. This means that the Settings app understands that they’re document and displays them with the correct “file” name. Unfortunately then the app can’t see the documents properly. Specifically, you can search for the document by name — which is nice — but you can’t actually open it. It’s odd and I’m not entirely sure what goes wrong, but when you call openWithCompletionHandler: it never actually calls the completion handler and the document never becomes available for use. (I initially thought that I just wasn’t being patient enough. I later accidentally left the app running for over two hours and the completion handler still never ran. That convinced me that it wasn’t a patience thing.)

I convinced myself that this couldn’t possibly be how it was supposed to work. My question on Stack Overflow didn’t get any answers and a reply to another, similar question said that the behaviour was “by design.”

I think it’s just broken and that no one is really using it. Either that or there’s a secret handshake that you have to give to the right person at WWDC.

So what’s shipping? Pretty much none of the above. I read about the new iOS 7 methods of Core Data – iCloud syncing and liked what I saw. Additionally iOS 7 comes with a few more APIs with better error handling hooks. This made me happier shipping with Core Data syncing. Just using “raw” Core Data meant that I could simplify a lot of code, use NSFetchedResultController and remove a bunch of the more fragile iCloud code (mostly NSMetadataQuery). While I was quite pleased with the KVO code I wrote, having far less code overall is a much bigger win.

And, in case you were wondering, I also de-scoped the Mac app and integrated Dropbox support instead.

But the bottom line is that this is all a lot harder than it should be. My concerns originally were around error conditions — something that has been very weak in iCloud since it’s introduction in iOS 5 — but it quickly morphed into questions about how it’s all supposed to work at all.

I once quipped on Twitter that you can tell which APIs are the ones Apple uses themselves as they’re the well designed ones. I think it’s pretty clear that Apple don’t use UIManagedDocument.