The story so far

Last year Yahoo! announced, with no notice, a significant change that had far reaching consequences for all third party applications including my iPhone program, Yummy. This is the third in a series of posts that discusses how I dealt with it.

On Monday I gave an overview of the problem and yesterday I looked at how those changes were announced and why they were tricky. Today we’ll look at how I actually implemented those technical details — though not at the code level so don’t worry if you’re not a programmer!

Open Source OAuth

The two leading contenders are OAuthConsumer and MPOAuth. It’s only fair to note that my evaluation of both projects was in October and November 2009. Progress is likely to have been made on both projects since then.

I managed to get the sample projects for both up and running pretty quickly. However on trying to log in I immediately came across a problem. Yahoo! displayed a big warning message suggesting that you don’t authorise any new applications for security reasons. And, worse, that this method of authenticating was going to be removed in a few months.

Even if I could get it working this was going to be a show-stopper. I couldn’t in clear conscience ask users to ignore such a prominent message. It took a long time pouring over the code and the OAuth specification to realise that what changed was the OAuth specification itself.

As the error message noted, due to a security flaw in OAuth 1.0 they issued an update to the specification (called 1.0a) and it was this that Yahoo’s servers was objecting to. Neither library had been updated to the new version.

The change to both projects was small and straight-forward but I was worried that I actually needed to make any alterations at all. Were they being maintained? Was anyone using them?

It fairly quickly became clear that both projects were really optmised for connecting to, or at least developed against, Twitter. As with all standards, each implementation has their own peculiarities and specialisations. To be clear, there is nothing wrong with this. All code has its own axioms and its intended use defines its “shape.”

When I first looked at it, one of the things I liked about MPOAuth was that it did a lot of the hard work for you. It kept an internal state machine so that it knew which message it should be sending to the authentication server and what it should expect to see in return. I’m all for other people doing the work for me. As long as you followed the “correct” order everything worked just fine.

However, in a real world application you can’t expect users to do the right thing every time; there needs to be some way to recover from errors or allow people to change their minds. And, try as I might, I couldn’t quite figure out how to force MPOAuth to do what I needed. In trying to be clever it appeared to be making my life harder rather than easier.

OAuthConsumer got closer to being an integral part of Yummy. The way it’s structured was much more closely aligned with how I would have done it but I came across endless errors trying to connect to Yahoo’s servers. Most came back with very obscure errors and I spent a huge amount of time single-stepping (debugging) the framework — something that I’d hoped I could avoid by taking pre-existing code.

To be fair, much of this comes down to OAuth itself. On the server side there is no way to determine which parameter is incorrect, only that the request is not internally consistent.

OAuth in Yummy

Having spent a lot of time trying to understand why the existing libraries did not work with Yahoo! I had accumulated a good understanding of the OAuth 1.0a specification but a much less clear understanding of the frameworks themselves. There was something about the way that they had both been put together that didn’t quite mesh with either my way of thinking or the way that I was trying to implement it in Yummy — I’m still not quite sure which.

So I decided to go ahead and make my own OAuth library. In some respects this was quite a risky move as it was going to be difficult code to write and it would be in a very fundamental part of the software, traditionally somewhere that I try to use simple, well tested code.

It’s difficult to say whether this was the quickest option. Maybe I could have changed one line in, say, OAuthConsumer and got everything working in half the time. But the amount of time that I spent struggling with them suggests not and this way I have full control over and full understanding of every line of code ((I’m also geeky enough to like writing this kind of code. So shoot me.)).

I ended up writing about four hundred lines of new code instead of using MPOAuth or OAuthConsumer. This is some reasonably low-level code and was tricky to debug as the server-side typically responded with a simple “invalid signature” error no matter what the actual problem was. (This is not to complain about Yahoo’s service. It’s just the nature of the beast, unfortunately.) However, once written it’s proven to be surprisingly resilient ((I’m only aware of one bug caused by the new code, and that problem, strictly speaking, was already present in Yummy, it just did not exhibit the same behaviour previously.)).

Coming tomorrow

Having decided on the low-level technical details I discovered that there was still a lot of work to do. That’s what I’ll be talking about tomorrow.

The story so far

Last year Yahoo! announced, with no notice, a significant change that had far reaching consequences for all third party applications including my iPhone program, Yummy. This is the second in a series of posts that discusses how I dealt with it.

Yesterday I spoke at a high level about my iPhone application and some changes that Yahoo! made to their side of the system. Today I’d like to talk in a little more detail about how those changes were announced and why they were tricky.

The announcement

On 19 October 2009, Yahoo! posted a blog announcing that all new accounts would be accessed with a Yahoo! ID.

As far as I know, this is the first time that anyone outside Yahoo! knew about this change. And the change took effect immediately.

At the time of the announcement, Apple were taking around two weeks to approve updates to iPhone applications. This means that even if it took no time at all to make the required changes — which as we’ll see is far from the case — it would still mean that a user of Delicious.com would have approximately two weeks to wait before any third party application would support their new scheme.

I don’t necessarily expect Yahoo! to support me personally but I do expect them to act in ways that are beneficial to their own users. Knowing that no third party applications would work with new accounts clearly does not meet this goal.

Maintaining a healthy ecosystem around their own software with well maintained, documented and supported APIs seems to work to their advantage, yet they did not do this. Just look at the flourishing market for Twitter clients for an example of it done well.

But, put bluntly, they failed.

It took over a month to update the API documentation. And when they did they simply added a one paragraph note saying that authentication can now be performed using OAuth. Until that point the only official resources were a few responses in the Delicious discussion forums.

Now, months after the change went live, the documentation is still barely adequate to code against. I think the lack of other third party clients that support Yahoo! ID authentication reinforces this point. (At the time of writing, I think I’m right in saying that Yummy and Yummy Browser are the only two iPhone clients that support Yahoo! ID logins.)

Unfortunately the haphazard approach to developers was only the first part of the problem.

Surely it’s just a different username and password?

You’d think that using a Yahoo! password rather than a dedicated Delicious password would be pretty easy, but you’d be wrong.

OAuth, as it’s called, doesn’t work like that.

The idea is that you shouldn’t have to trust anyone other than Yahoo! with your Yahoo! password. So, bizarrely, I had to make Yummy work without ever knowing what your username or password is!

Without getting too bogged down in detail, this is roughly how it works:

1. The client launches a web browser. Since an iPhone can’t multi-task third-party applications, this means exiting and switching over to Safari ((The iPhone OS does allow applications to include an embedded web browser, indeed Yummy already has one to allow users to preview their bookmarks. However, the OAuth specification recommends against this approach for authentication. The idea is that you should only trust Yahoo! with your username and password and that entering them into a third-party application might break that trust. An early version of the change did include an embedded browser and I may yet revert back to that version.)). The changes announced for iPhone OS 4.0 don’t help here
2. The user logs into Yahoo! The server responds with a code
3. The user launches the client again and enters the code

Then every time the client wants to access a protected resource it does some cryptographic magic using the code (generating what’s called a “hash”) and sends that along with the request.

But the devil, as ever, is in the detail. The detail in this case is a twelve page document that assumes that you know what things like “octet string, first base64-encoded” means. Therefore in practice you end up looking up every second paragraph which slows things down.

Of course, writing the code yourself would be the hard option. For a standard such as OAuth there are bound to be some Open Source libraries that can be used without too much trouble.

Next…

Tomorrow I’ll talk about those Open Source libraries and what I ended up using.

Background

For the benefit of any new visitors, I develop an iPhone application called Yummy that connects to the Delicious.com “social bookmarking” website, allowing you to access and share your bookmarks using a far more usable interface than the native web or mobile optimised version of the site.

In the olden days — that is more than six months ago — to access Delicious all you had to do was enter a user name and password. As a developer it was very similar. Whenever you wanted to access something that required authorisation — adding a new bookmark for example — you had to supply the same credentials as you would on the web: a username and password. This, in web terminology, is called “basic authentication.”

And then in October last year, Yahoo! announced that any new accounts would be linked to their Yahoo! ID, the same username and password that are used to protect Yahoo! Mail, Flickr and most other Yahoo! services. A month later they allowed existing users to transition their old account to the new Yahoo! ID world.

On the Delicious.com website things look very similar. You still enter a username and password and can then access protected resources. But for developers things work a bit differently.

In the rest of this series of posts I’d like to talk about this transition and the difficulties involved. In the six months that it’s taken me to get around to writing this I’ve calmed down a lot but I’m sure you’ll be able to feel the frustration I felt at some points.

I’m going to split the discussion into a number of different “chapters.” First there’s how Yahoo! dealt with the transition. Next are the technical challenges, the options I tried and, eventually, the solution I settled on.

What follows is not densely technical, in the sense that I’m not going to be talking to the level of individual lines of code. However many of the decisions and trade-offs are somewhat technical in nature. Most people should be able to follow the process if not every last detail.

Tomorrow we’ll start with the announcement and why what they changed made things tricky for all third-party clients, including Yummy.