Tag Archives: Opinion

Opinion

Delicious Debrief (Part 3/5)

The story so far

Last year Yahoo! announced, with no notice, a significant change that had far reaching consequences for all third party applications including my iPhone program, Yummy. This is the third in a series of posts that discusses how I dealt with it.

On Monday I gave an overview of the problem and yesterday I looked at how those changes were announced and why they were tricky. Today we’ll look at how I actually implemented those technical details — though not at the code level so don’t worry if you’re not a programmer!

Open Source OAuth

The two leading contenders are OAuthConsumer and MPOAuth. It’s only fair to note that my evaluation of both projects was in October and November 2009. Progress is likely to have been made on both projects since then.

I managed to get the sample projects for both up and running pretty quickly. However on trying to log in I immediately came across a problem. Yahoo! displayed a big warning message suggesting that you don’t authorise any new applications for security reasons. And, worse, that this method of authenticating was going to be removed in a few months.

Even if I could get it working this was going to be a show-stopper. I couldn’t in clear conscience ask users to ignore such a prominent message. It took a long time pouring over the code and the OAuth specification to realise that what changed was the OAuth specification itself.

As the error message noted, due to a security flaw in OAuth 1.0 they issued an update to the specification (called 1.0a) and it was this that Yahoo’s servers was objecting to. Neither library had been updated to the new version.

The change to both projects was small and straight-forward but I was worried that I actually needed to make any alterations at all. Were they being maintained? Was anyone using them?

It fairly quickly became clear that both projects were really optmised for connecting to, or at least developed against, Twitter. As with all standards, each implementation has their own peculiarities and specialisations. To be clear, there is nothing wrong with this. All code has its own axioms and its intended use defines its “shape.”

When I first looked at it, one of the things I liked about MPOAuth was that it did a lot of the hard work for you. It kept an internal state machine so that it knew which message it should be sending to the authentication server and what it should expect to see in return. I’m all for other people doing the work for me. As long as you followed the “correct” order everything worked just fine.

However, in a real world application you can’t expect users to do the right thing every time; there needs to be some way to recover from errors or allow people to change their minds. And, try as I might, I couldn’t quite figure out how to force MPOAuth to do what I needed. In trying to be clever it appeared to be making my life harder rather than easier.

OAuthConsumer got closer to being an integral part of Yummy. The way it’s structured was much more closely aligned with how I would have done it but I came across endless errors trying to connect to Yahoo’s servers. Most came back with very obscure errors and I spent a huge amount of time single-stepping (debugging) the framework — something that I’d hoped I could avoid by taking pre-existing code.

To be fair, much of this comes down to OAuth itself. On the server side there is no way to determine which parameter is incorrect, only that the request is not internally consistent.

OAuth in Yummy

Having spent a lot of time trying to understand why the existing libraries did not work with Yahoo! I had accumulated a good understanding of the OAuth 1.0a specification but a much less clear understanding of the frameworks themselves. There was something about the way that they had both been put together that didn’t quite mesh with either my way of thinking or the way that I was trying to implement it in Yummy — I’m still not quite sure which.

So I decided to go ahead and make my own OAuth library. In some respects this was quite a risky move as it was going to be difficult code to write and it would be in a very fundamental part of the software, traditionally somewhere that I try to use simple, well tested code.

It’s difficult to say whether this was the quickest option. Maybe I could have changed one line in, say, OAuthConsumer and got everything working in half the time. But the amount of time that I spent struggling with them suggests not and this way I have full control over and full understanding of every line of code ((I’m also geeky enough to like writing this kind of code. So shoot me.)).

I ended up writing about four hundred lines of new code instead of using MPOAuth or OAuthConsumer. This is some reasonably low-level code and was tricky to debug as the server-side typically responded with a simple “invalid signature” error no matter what the actual problem was. (This is not to complain about Yahoo’s service. It’s just the nature of the beast, unfortunately.) However, once written it’s proven to be surprisingly resilient ((I’m only aware of one bug caused by the new code, and that problem, strictly speaking, was already present in Yummy, it just did not exhibit the same behaviour previously.)).

Coming tomorrow

Having decided on the low-level technical details I discovered that there was still a lot of work to do. That’s what I’ll be talking about tomorrow.

Opinion

Delicious Debrief (Part 2/5)

The story so far

Last year Yahoo! announced, with no notice, a significant change that had far reaching consequences for all third party applications including my iPhone program, Yummy. This is the second in a series of posts that discusses how I dealt with it.

Yesterday I spoke at a high level about my iPhone application and some changes that Yahoo! made to their side of the system. Today I’d like to talk in a little more detail about how those changes were announced and why they were tricky.

The announcement

On 19 October 2009, Yahoo! posted a blog announcing that all new accounts would be accessed with a Yahoo! ID.

As far as I know, this is the first time that anyone outside Yahoo! knew about this change. And the change took effect immediately.

At the time of the announcement, Apple were taking around two weeks to approve updates to iPhone applications. This means that even if it took no time at all to make the required changes — which as we’ll see is far from the case — it would still mean that a user of Delicious.com would have approximately two weeks to wait before any third party application would support their new scheme.

I don’t necessarily expect Yahoo! to support me personally but I do expect them to act in ways that are beneficial to their own users. Knowing that no third party applications would work with new accounts clearly does not meet this goal.

Maintaining a healthy ecosystem around their own software with well maintained, documented and supported APIs seems to work to their advantage, yet they did not do this. Just look at the flourishing market for Twitter clients for an example of it done well.

But, put bluntly, they failed.

It took over a month to update the API documentation. And when they did they simply added a one paragraph note saying that authentication can now be performed using OAuth. Until that point the only official resources were a few responses in the Delicious discussion forums.

Now, months after the change went live, the documentation is still barely adequate to code against. I think the lack of other third party clients that support Yahoo! ID authentication reinforces this point. (At the time of writing, I think I’m right in saying that Yummy and Yummy Browser are the only two iPhone clients that support Yahoo! ID logins.)

Unfortunately the haphazard approach to developers was only the first part of the problem.

Surely it’s just a different username and password?

You’d think that using a Yahoo! password rather than a dedicated Delicious password would be pretty easy, but you’d be wrong.

OAuth, as it’s called, doesn’t work like that.

The idea is that you shouldn’t have to trust anyone other than Yahoo! with your Yahoo! password. So, bizarrely, I had to make Yummy work without ever knowing what your username or password is!

Without getting too bogged down in detail, this is roughly how it works:

  1. The client launches a web browser. Since an iPhone can’t multi-task third-party applications, this means exiting and switching over to Safari ((The iPhone OS does allow applications to include an embedded web browser, indeed Yummy already has one to allow users to preview their bookmarks. However, the OAuth specification recommends against this approach for authentication. The idea is that you should only trust Yahoo! with your username and password and that entering them into a third-party application might break that trust. An early version of the change did include an embedded browser and I may yet revert back to that version.)). The changes announced for iPhone OS 4.0 don’t help here
  2. The user logs into Yahoo! The server responds with a code
  3. The user launches the client again and enters the code

Then every time the client wants to access a protected resource it does some cryptographic magic using the code (generating what’s called a “hash”) and sends that along with the request.

But the devil, as ever, is in the detail. The detail in this case is a twelve page document that assumes that you know what things like “octet string, first base64-encoded” means. Therefore in practice you end up looking up every second paragraph which slows things down.

Of course, writing the code yourself would be the hard option. For a standard such as OAuth there are bound to be some Open Source libraries that can be used without too much trouble.

Next…

Tomorrow I’ll talk about those Open Source libraries and what I ended up using.

Opinion

Delicious Debrief (Part 1/5)

Background

For the benefit of any new visitors, I develop an iPhone application called Yummy that connects to the Delicious.com “social bookmarking” website, allowing you to access and share your bookmarks using a far more usable interface than the native web or mobile optimised version of the site.

In the olden days — that is more than six months ago — to access Delicious all you had to do was enter a user name and password. As a developer it was very similar. Whenever you wanted to access something that required authorisation — adding a new bookmark for example — you had to supply the same credentials as you would on the web: a username and password. This, in web terminology, is called “basic authentication.”

And then in October last year, Yahoo! announced that any new accounts would be linked to their Yahoo! ID, the same username and password that are used to protect Yahoo! Mail, Flickr and most other Yahoo! services. A month later they allowed existing users to transition their old account to the new Yahoo! ID world.

On the Delicious.com website things look very similar. You still enter a username and password and can then access protected resources. But for developers things work a bit differently.

In the rest of this series of posts I’d like to talk about this transition and the difficulties involved. In the six months that it’s taken me to get around to writing this I’ve calmed down a lot but I’m sure you’ll be able to feel the frustration I felt at some points.

I’m going to split the discussion into a number of different “chapters.” First there’s how Yahoo! dealt with the transition. Next are the technical challenges, the options I tried and, eventually, the solution I settled on.

What follows is not densely technical, in the sense that I’m not going to be talking to the level of individual lines of code. However many of the decisions and trade-offs are somewhat technical in nature. Most people should be able to follow the process if not every last detail.

Tomorrow we’ll start with the announcement and why what they changed made things tricky for all third-party clients, including Yummy.

Opinion

Communication

“Do you ever change this type of trade?”

I was sat on the trading floor discussing a new feature that I was implementing with the person who would be using it the most.

“No, never.”

This was one detail of the change that would have far-reaching consequences in the code. A “no” would mean a few days of development, a “yes” would indicate several weeks.

“Are you absolutely sure? You don’t change it even once a month?”

I knew they’d like the smaller estimate but, equally, I knew that I didn’t want to end up trying to implement several weeks worth of functionality in the smaller time frame. I’ve seen that kind of thing happen too often.

“John ((Names have been changed to protect the… well, they work for a bank so I hesitate to say “innocent” but you know what I mean.)), we don’t ever change these trades do we?”

John was the head trader. If anyone would know it would be him.

He rubs his head and leans back, thinking.

“Why would you want to? No, I can’t see us ever changing one.”

Of course, you can guess what happened thirty minutes into the first trading day with the new software.

So, what happened here? Why did we get the requirements so wrong? And why did we only find the mismatch after the system was moved into production?

Both the problem and the solution is the same thing: communication. Or more precisely, communicating using the right language.

Some developers are happy to only “speak” technical, proud that they are masters of their programming environment but ignorant of their users problems or how they really use the software.

Above I started out correctly, I was trying to understand the traders business and talking in terms of booking trades, positions, legal entities and a bunch of acronyms that would make even less sense out of context.

But I made one error: I used the word “change” without defining what I meant. I meant, well, any change. And so did they. Yet they didn’t consider moving a trade from one book to another to be a change and, unfortunately, I did.

You’d like to think that there were checks and balances in place to make sure that this kind of thing didn’t happen. And there were. In addition to informal and formal testing, there was over a week of “parallel running,” where the traders had to use the old and the new system together and check that the results were the same in both of them.

Were there any moved trades during this time? Of course. Why did the traders not notice? Well, it was about right; the differences, while present, were explicable and so not considered significant enough to mention even though I asked to hear about any problems at all.

So, again, communication. Or at least human nature. I wanted to hear about any differences but they tried to help me by only talking about differences that they couldn’t account for.

What’s the answer? Well, I’m not sure there’s an easy one. “Understanding your user” is a short, simple phrase but hides so much. If you spent the time to fully understood their job you probably wouldn’t have the time to do your own. But finding the balance is crucial.

Links

My delicious.com bookmarks for April 20th through April 25th

  • No sense of humour at all – "Personally, I'd love to see the Pope in a debate on abortion, where he would actually have to address difficult questions and defend his own ideas. Best idea yet would be a debate on various controversial topics, like birth control, abortion, the role of women in the church, and homosexuality…with the Pope on one side, and Stephen Fry on the other. It could be perfectly respectful, and it would be hilarious."
  • Audio slideshow: Hubble’s first 20 years – Amazing. Beautiful.
  • Do liberals read only liberal blogs? – The dangers of the "long tail…" I deliberately go read Daily Mail headlines on a regular basis just to check that I'm sane. The moment I start to agree, please shoot me.
Opinion

The Up-Sell

I don’t mean to single out a single business here. The flaw I’m pointing out is shared by many sites but this post was inspired by a recent visit to TripIt. In general it’s a great service. It’s well thought out, allowing you to enter all your details with a minimum of effort; just forwarding your email confirmation to them is a masterstroke.

However. (You knew that was coming.) However, many links on the main page are non-functional, by which I mean they push you straight through to their paid-for service sign-up form.

The “tricky” part is that before you press them it is difficult to know which links actually work and which ones just ask for money.

There are a number of other tricks that some sites have. Another favourite is the interstitial screen, forcing you to view adverts before you can do what you actually want to do.

But it’s not just that I find it obnoxious. I don’t have data but I do have a nice anecdote that shows that it doesn’t really work.

During the dot.com boom I helped build a website. The launch went pretty well but the client decided that they wanted to push a secondary product, one with great margins but where customers really needed to be vetted. (I don’t want to get into specifics but it was a financial product.) The marketing people said that a pop-up would be the right thing to do.

We balked at the idea. At the time, pop-ups were the scourge of the Internet. They were used on all the least reputable sites. Technically adept users closed them without looking; the less fortunate were conned into either filling their screen with pointless adverts or visiting website they had no interest in. Pop-up blockers were a few years away.

In short, we felt that at best there was a reputational risk. Unfortunately we couldn’t come up with numbers to show that it was financially a bad idea, plus it was actually pretty cheap to implement. So they asked us to go ahead with it, over our objections.

As I recall it didn’t last very long.

After go-live there was a substantial up-tick in the number of people applying for this secondary product. However, there was actually a drop in the number of people who were accepted. That’s to say that it attracted exactly the wrong kind of person, which is bad enough, but there was also a cost associated with each rejected application.

Moving back to 2009, I think the problem with pushing your paid products too hard is that you actually make your free version less appealing. And, frankly, if your free version is a pain to use I’m certainly not going to pay for the full version just to make the evil bits go away.

To be clear, I have nothing against the so-called “freemium” business model. It can work really well. Flickr, for example, seems to have the balance about right: the site is useful even if you don’t pay for it with the extras useful for regular users. And paying LWN readers can get their content a week ahead of other people.

In short, if your paid extras are genuinely useful you don’t need to be obnoxious, you don’t need lots of “dead” links or interstitial adverts. And making your free version painful is most certainly not the answer.