Category Archives: Programming

Articles about programming.

Starting with CloudKit Syncing

One of the slightly more hidden features of CloudKit, Apple’s cloud-based back-end service for applications, is that you can use it to synchronise content as well as simply query it. I use this approach to sync favourite search terms between devices in my app Yummy.

However, I found the process wasn’t as well documented as it might be. It’s all there, but it’s written as many man pages are: it makes total sense if you already know what you’re looking for. This post is my attempt to make the process clearer.

We’ll start by inserting some data into the cloud data store and then try to get it back out again.

Pushing data to CloudKit is pretty straight forward. As we’ll see later there’s a little more preparation required in practice, but at it’s simplest it looks like this:

// Get a reference to the private cloud database
CKDatabase* db = [[CKContainer defaultContainer] privateCloudDatabase];

// create a new record
CKRecord* myRecord = [[CKRecord alloc] initWithRecordType:@“Record” zoneID:zoneID];

// set the interesting values
[myRecord setObject:@“Data” forKey:@“Key”];

[db saveRecord:myRecord completionHandler:^(CKRecord* record, NSError* error) {
  // error handling
  // save a local copy in cache
}];

We’ll come back to the zoneID variable used in the second line.

It’s asynchronous and only takes a couple of API calls. Nice.

Once data has been pushed to the cloud you have to get it back again. In my case there are few enough records that I could just download everything and do a local “diff.” That would look like this:

// create a query
CKQuery* query = [[CKQuery alloc] initWithRecordType:@“record”
                                              predicate:[NSPredicate predicateWithValue:YES]];

// define what to do with the records to be returned
CKQueryOperation* queryOperation = [[CKQueryOperation alloc] initWithQuery:query];
queryOperation.recordFetchedBlock = ^(CKRecord* record) {
  // do something with each record
};

// fire off the query
[db addOperation:queryOperation];

Doing the local diff is left as an exercise for the reader.

Again, just a few lines of code. (Quick aside: I’m not sure if it’s a good thing that the block is called for every record individually. It feels like it might be easier given a block of records, but CloudKit is generally consistent about this one record at a time approach.)

But that feels like it’s missing the point. CloudKit supports just getting a list of what’s changed since the last sync.

The class to find the records that have changed since the last sync is CKFetchRecordChangesOperation, but if we put that in a method:

CKFetchRecordChangesOperation* changesOperation = [[CKFetchRecordChangesOperation alloc] initWithRecordZoneID:[CKZoneID defaultZone]
previousServerChangeToken:nil];

[[[CKContainer defaultContainer] privateCloudDatabase] addOperation:changesOperation];

(The nil token means we’re starting from scratch — get us everything. Next time we sync, we pass in the token given at the end of the last sync.)

However if we run it we get an error:

<CKError 0x7fd326282bb0: “Server Rejected Request” (15/2027); server message = “AppDefaultZone does not support sync semantics”; uuid = 00138311-1714-4651-BE36-9D85B3CB5D51; container ID = “containerID”>

So, in short, the default configuration does not allow us to sync! We have to create a new record zone to store our records. A corollary of this is that you can only sync private stores, since you cannot create new zones for the public cloud database.

In principle this is quite simple but in practice it can get a little hairy as every call to CloudKit is asynchronous and there’s lots of error handling to consider.

I’m not going to show all my working (because what I have is pretty hacky and I’m sure you can do better) but the algorithm is:

  1. Are we logged into iCloud? If not, give up now
  2. Do we have a record of the current zone ID? If so, jump to step 7
  3. Fetch the list of zones that we can use
  4. Loop through the list returned. If there’s one we recognise, save the reference and jump to step 7
  5. Create a new zone
  6. Save the Zone ID in the completion handler
  7. Now get on with our sync / insert / update / delete operation

Note that queries tend to pick the right zone automatically, so if you’re searching rather than syncing you don’t need to worry about this.

And that’s it. Syncing is possible, just a little convoluted.

Moving an app from Paid to Free

I’ve seen quite a few people saying that it isn’t possible to move an iOS app from paid to being free with an in-app purchase to unlock the full functionality. Fortunately they’re wrong.

“Traditionally” I would have had to remove version one from sale and offer a completely new app, which would have meant that existing users would have to pay again to get the same functionality. Or I’d have to support two apps. Or I’d keep the same app in the store and all existing users would get downgraded to the free version. None of these solutions seemed fair to existing users.

What I wanted was for people who had bought version one to get the full, unlocked version and for new users to be promoted for the paid upgrade.

Since iOS 7 came out in 2013 that it entirely possible. I’ll explain how it’s done here. This isn’t just some theoretical “I’ve seen the documentation” claim – I’ve done it with one of my own apps, Rootn Tootn.

The really short answer: take a look at the session 308 video from WWDC 2013. That’s the only information from Apple that explains how to do it. They have documented the API calls that are required but the actual process is left as an exercise for the interested student. And there are quite a few steps if you want to do it properly.

Firstly you need to get the app receipt. Before iOS 7 this only made sense for IAP but now they are available for all purchases and come in the same format as receipts from the Mac App Store.

Receipts have a number of useful features. In the past they have been used to validate purchase, and they can still be used for this. What’s interesting with the new receipts is that they include both the original purchase and the version number of that original purchase. This means that we can decide whether a user gets the paid functionally by looking for either an in app purchase or a purchase date before a particular time or, more likely, before a particular version.

When you download an app you should get a receipt automatically but you can also use the SKReceiptRefreshRequest class to force one to be generated. (This is also useful during development where, obviously, there is no receipt.)

Once the refresh has completed, you use [NSBundle appStoreReceiptURL:] to access the receipt.

Once you have the receipt the bad news starts.

It’s not in a user friendly format. And Apple do not provide any APIs to read it. Check out Apple’s documentation:

The outermost portion (labeled Receipt in the figure) is a PKCS #7 container, as defined by RFC 2315, with its payload encoded using ASN.1 (Abstract Syntax Notation One), as defined by ITU-T X.690. The payload is composed of a set of receipt attributes. Each receipt attribute contains a type, a version, and a value.

If security is important to you, you should probably write your own code to do this. ASN.1 is a standard format and it’s not that hard.

There are apps that generate the validation code, such as Tighten Pro and Receigen. I can’t vouch for either of them but the reviews seems positive.

There are also Open Source projects that do the same thing. I’ve used RMStore; there’s also VerifyStoreReceiptiOS. The main disadvantage of these is that, as standard, open code it makes it easier for crackers to reverse engineer how you remember that a purchase has been made.

And there you have it. It is possible. It’s just a lot harder than you might imagine. Remember this when someone tells you that it can’t be done.

Swift Types

If you look at the Swift Language guide, you get the distinct impression that the type system is sleek and modern. However the more you dig into it the more eccentricities you find.

The one I’m going to look at today makes sense only if you look at the problem domain from a slightly skewed perspective. I’ve been trying to think whether this is a sensible, pragmatic way of designing a language or a mistake. Judge for yourself.

So, the feature. Let’s define a dictionary:

var test1 = [ "Foo" : "Bar" ]

Check the type and we find that it’s of type Dictionary<String,String>. The generics and type inference are doing exactly what you’d image.

test1["Test"] = "Works"

So basically it’s all good.

So, what type is this expression?

var test2 = [:]

And why does this not work?

test2["Test"] = "Doesn't work"

Let’s take a step back. What’s the problem? Well, [:] is an empty dictionary but give us no clue what the type is. Remember, Swift dictionaries and arrays use generics, so the compiler only allows objects of a particular type to be added.

A good guess for the type would be Dictionary<AnyObject,AnyObject>. But a little fishing around tells you that’s not the case because AnyObject is neither “Hashable” or “Equatable” and keys need to be both.

The answer? test2 is an NSDictionary. That is, in this one circumstance, Swift extends outside its native dictionary type and decides to use a class found in Foundation.

Once you know that, it is clear that the second line should be:

test2.setValue("Does work now", forKey:"Test")

Maybe if you’re familiar with the guts of both Objective C and Swift this behaviour makes sense, but a language built-in returning a completely different type just because it can’t figure out the type feels broken to me.

In the end I think I’ve convinced myself that, while it might be convenient to allow this syntax, it’s a bad idea to saddle the language with these semantics so early on. In a few years when no one uses Objective C or when Swift is no longer fully tied to Cocoa, will this make sense?

I would prefer to see it being a compiler error with the correct approach being explicit with the type:

var test2:Dictionary<String,String> = [:]

Thoughts?

Learning Swift

Swift is a new programming language designed by Apple for development on OS X and iOS. I thought that I should try to learn it a little so I decided to convert a non-trivial collection of classes from one of my apps (www.cut) into Swift. I always find it better to work on a real project rather than just to play around with things aimlessly. Also, by re-working an old project, I knew that all the problems I would find would be language related rather than anything to do with the architecture.

The classes are also data related rather than being UI, so it is mostly a test of the language itself rather than how it interfaces with Objective C.

First impressions are good. Swift is mostly nice and consistent, which although sounding like damning with faint praise, is actually a compliment. I read a little of the language guide and dove straight in. A lot of the attempts to get the following right were me just typing stuff, guessing the syntax rather than looking it up.

Quite by accident I think my code sample inadvertently shows an area of strength for Objective-C and weakness for Swift.

The idea of the code is that it reads a Plist, instantiates a class based on that configuration and fills in a number of properties.

The first half of the code looks like this:

        NSError* error = nil;
        NSString *plistPath = [[NSBundle mainBundle] pathForResource:@"XXX" ofType:@"plist"];
        NSData *plistXML = [[NSFileManager defaultManager] contentsAtPath:plistPath];
        NSDictionary *temp = (NSDictionary *)[NSPropertyListSerialization propertyListWithData:plistXML
                                                                                       options:NSPropertyListMutableContainersAndLeaves
                                                                                        format:NULL
                                                                                         error:&error];
        NSArray* values = [temp objectForKey:@"Entries"];

This was pretty straight forward to convert into Swift, though the type system gave me issues:

    let plistPath =  NSBundle.mainBundle().pathForResource("XXX", ofType: "plist")
    let plistXML = NSFileManager.defaultManager().contentsAtPath(plistPath)

    var error:NSError? = nil
    var format:CMutablePointer<NSPropertyListFormat>? = nil
    let immutable:NSPropertyListReadOptions = 0
    var pList = NSPropertyListSerialization.propertyListWithData(plistXML,
        options:NSPropertyListReadOptions(NSPropertyListMutabilityOptions.Immutable.toRaw()),
        format:format!,
        error: &error) as NSDictionary

Getting the options property seems very clumsy; I’m sure that there must be a better way of doing it.

I had a real problem with the in/out parameters format and error. Not only was the documentation confusing but the Swift Playground kept crashing making it difficult to distinguish between what I was doing wrong and where the compiler itself was messing up. It’s also a bit odd that, though both are in/out parameters, that they both need different methods to extract the values.

(To be fair, this is a beta and it is the first version of a whole language and compiler. I mention the crashes not because they’re unexpected or even especially bad, just as an honest description of the difficulty I had.)

The next section, using the data in the plist, was much more problematic. The code looks like this, but I’ve trimmed a lot so what we have here isn’t terribly useful now!

        proxy = nil;
        for (NSDictionary* i in values) {
            if ([thing isEqualToString:[i objectForKey:@"Class"]]) {
                dynamicClass = NSClassFromString([i objectForKey:@"BaseClass"]);
                proxy = [[dynamicClass alloc] init];
            }
        }

I didn’t do the straight conversion. A few more years of Cocoa programming allowed me to notice an optimisation:

    let valueList = values.filteredArrayUsingPredicate(NSPredicate(format: "Class = %@", value))
    let valueData = valueList[0] as NSDictionary

This same approach would work in Objective-C.

Next I tried:

  let dynamicClass = NSClassFromString(valueData["BaseClass"])
  let proxy = dynamicClass()

The first line works as expected. The second line doesn’t compile.

Is there anything else that we can do with dynamicClass? Let’s see. It’s an AnyClass which is a type alias for AnyObject.Type. Which doesn’t really help.

I tried casting it to a base class but no matter what I tried I couldn’t alloc/init it (in Objective C terms).

Josh Smith figured out how to do it by creating a factory class in Objective-C.

I tried (and failed) to get it to work by calling some of the Objective-C runtime directly:

var bytes:Byte[] = [0,0,0,0]
let b = objc_constructInstance(a, &bytes)

But the second line doesn’t work when using ARC. (To be fair, Xcode struck through the definition so I didn’t have much confidence that it would work!)

So that leaves Josh’s call out to Objective-C to be the best method that I’m aware of.

In the end I just used a switch statement to select between the relatively limited number of options that I had to choose between. Not as clever, but maybe that’s a good thing?

What you forgot from your Computer Science Degree

Last night I did a short presentation about my WSLHTMLEntities open source project at the London iOS Developer Group meeting. You can see the slides here:

Since last time I did a talk there people snickered because I built the slides using PowerPoint, this time I decided to use the latest Apple technology: Keynote in iCloud. Unfortunately this was a bit too new for the Mac Pro they use in the Apple Store, so we ended up downloading a copy in PowerPoint format and loading that into the local copy of Keynote. Nothing is ever simple.

One question I got at the end that I was unable to answer is how well it performs compared to other solutions.

This afternoon I ran a few quick tests.

Firstly I added two further methods of performing the HTML entity replacement:

  • Google Cocoa Toolbox
  • Use the new iOS 7 NSAttributedString feature where it can load a HTML document

The last option, while built-in and easy to implement, is not something you’d want to consider if you had any performance requirements. First, it has to run on the main thread. Second, it’s memory footprint is way higher than either of the other two solutions. And, finally, it’s slow. In my tests I did a loop of 10,000 iterations. Because of the memory problem I only ran 1000 iterations of the NSAttributedString solution, but it was still about 15 times slower than my version when doing the full 10,000 records (i.e., over 150 times slower overall).

So I’ve not included it in the charts below.

HTMLEntityChart

I ran eight simple tests with each of the parsers.

There is no clear winner.

As I speculated, WSLHTMLEntities is much more consistent than the other two. Google’s varies a lot depending on where in the mapping list the “solution” lies. Replacing &loz; (near the end of the list) takes 50% longer than &amp; (near the beginning) for example.

Still, you’d have to be pretty obsessive to pick one of these solutions for performance reasons alone. Interesting to find out, though!

NSFetchedResultsController and iCloud

This took me a while to figure out so I thought it was worth blogging about. The short version: I’m using Core Data with iCloud syncing and it works… mostly. When starting up for the first time – when there is already data in iCloud –  none of the data appears in a table view, but restarting the app correctly displays it.

I know what you’re thinking: you’re not merging the updates into the right managed object context. Nope. Sorry. Thinking that was the problem is probably why it took me quite so long to track the real problem down!

So, what does the problem look like?

I set up my Core Data stack in the app delegate. Part of that includes connecting to iCloud:

    NSURL *storeURL = [[self applicationDocumentsDirectory] URLByAppendingPathComponent:@"XXX.sqlite"];
    NSDictionary *options = @{
                              NSMigratePersistentStoresAutomaticallyOption: @YES,
                              NSInferMappingModelAutomaticallyOption: @YES,
                              NSPersistentStoreUbiquitousContentNameKey : @"XXX"
                              };

    NSError *error = nil;
    _persistentStoreCoordinator = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:[self managedObjectModel]];
    if (![_persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType
                                                   configuration:nil
                                                             URL:storeURL
                                                         options:options
                                                           error:&error]) {

The app delegate passes the managed object context through to the main view controller which then uses it to create a NSFetchedResultsController:

    NSFetchRequest *fetchRequest = [[NSFetchRequest alloc] init];
    // Edit the entity name as appropriate.
    NSEntityDescription *entity = [NSEntityDescription entityForName:@"XXX" inManagedObjectContext:self.managedObjectContext];
    [fetchRequest setEntity:entity];

    // Set the batch size to a suitable number.
    [fetchRequest setFetchBatchSize:20];

    // Edit the sort key as appropriate.
    NSSortDescriptor *sortDescriptor = [[NSSortDescriptor alloc] initWithKey:@"someTimestamp" ascending:NO];
    NSArray *sortDescriptors = @[sortDescriptor];

    [fetchRequest setSortDescriptors:sortDescriptors];

    // Edit the section name key path and cache name if appropriate.
    // nil for section name key path means "no sections".
    NSFetchedResultsController *aFetchedResultsController = [[NSFetchedResultsController alloc] initWithFetchRequest:fetchRequest
                                                                                                managedObjectContext:self.managedObjectContext
                                                                                                  sectionNameKeyPath:nil
                                                                                                           cacheName:@"Master"];
    aFetchedResultsController.delegate = self;
    self.fetchedResultsController = aFetchedResultsController;

	NSError *error = nil;
	if (![self.fetchedResultsController performFetch:&error]) {
        // Replace this implementation with code to handle the error appropriately.
        // abort() causes the application to generate a crash log and terminate. You should not use this function in a shipping application, although it may be useful during development.
	    NSLog(@"Unresolved error %@, %@", error, [error userInfo]);
	    abort();
	}

(Again, this is pretty much Apple-standard boilerplate code –  nothing clever going on here.)

But what I found was that when there was already data in iCloud, the records did not automatically appear in the main view controller.

Eventually I found that this was not a threading issue – merging the changes into one context and trying to read them from another – by adding an explicit fetch request on a “debug” button. Doing this I could see the new data, even though the fetched result controller could not.

In my app delegate I listened for a number of notifications: NSPersistentStoreDidImportUbiquitousContentChangesNotification and NSPersistentStoreCoordinatorStoresWillChangeNotification. My expectation was that NSPersistentStoreCoordinatorStoresWillChangeNotification would fire before switching from the fallback store to the “real” one and NSPersistentStoreDidImportUbiquitousContentChangesNotification would fire when the new data was available.

I was half right. NSPersistentStoreCoordinatorStoresWillChangeNotification fired on a background thread, so I used GCD to ping it onto the main thread and reset the context.

But NSPersistentStoreDidImportUbiquitousContentChangesNotification didn’t fire at all. I guess the objects didn’t change as such, they just became available, which feels like a slightly false distinction to me.

So my next guess was to fire the fetch request again. The data was visible in the main threads context so surely this would find the data?

Nope. (And don’t call me Shirley.)

I was getting pretty lost at this point so I ended up just semi-randomly stopping the code and looking around in the debugger.

And, long story short, I realised that fetch requests have a reference to the persistent store in them – check out the affectedStores method. This meant that the NSFetchedResultsController was happily, and correctly, reporting on the empty and no longer used fallback store and completely ignoring the new and fully populated iCloud store.

The simple solution was to listen for the NSPersistentStoreCoordinatorStoresDidChangeNotification and create a completely new fetch request.

- (void)storesDidChange:(NSNotification*)notification {
    self.fetchedResultsController = nil;
    [NSFetchedResultsController deleteCacheWithName:@"Master"];
    [self.tableView reloadData];
}

(I did think of just adding the new persistent store to the old fetch request but I wasn’t sure that this would create the refresh anyway and, given the frequency with which this is likely to happen, I thought it would be cleaner just to start from scratch.)

These weird problems almost always boil down to just a couple of lines of code. This time was no exception.