Automated iOS Jenkins builds with application tests and Core Data

I recently started some iOS development with a new team and had the chance to set up a server for continuous integration builds. Since our teams are already familiar with Jenkins that seemed like the logical place to start, and the Xcode plugin available works reasonably well out-of-the-box for simply building the application. However, running tests – especially what Apple calls application tests (anything accessing a bootstrapped NSApplication through the simulator) using Core Data – took a little bit of tweaking. Here’s what was needed as of Xcode 4.2.

  1. Install the latest version of Jenkins, either on a Mac with Xcode installed or another server that has such a Mac available as a slave for any iOS builds
  2. Install a plugin for your source control system of choice if not already present (I’m using Git with the default Git plugin - considering the great Git support added in Xcode 4 with Git repos enabled by default, if you’ve been looking for an excuse to switch this is it!)
  3. Install the Xcode plugin
  4. Create a build following the plugin build instructions for a basic release build of the application, and especially pay attention to the keychain notes (I went ahead and entered the login password of the build account so that Jenkins could unlock it itself)
  5. Run a build and verify that everything is successful up to this point, and if not, troubleshoot
  6. Edit the build and add a separate build target for testing following the plugin test instructions, specifying iphonesimulator as the SDK (you can enter a full absolute path like the example if you wish, but simply iphonesimulator should find the latest), and make sure it points to your project’s test target instead of the primary build target (e.g. MyAppTests instead of MyApp)
  7. Run a build and look for errors. Assuming you have one or more application tests expecting to run in the iOS simulator it will probably have skipped all the tests with an error like ”Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).”
  8. To fix the error above (or prepare for it, if application tests don’t exist yet but will eventually), on the Jenkins server (or slave if running the iOS build on a slave) edit:
    /Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/RunPlatformUnitTests (if Xcode <= 4.2)
    /Applications/ (if Xcode >= 4.3)

    and change line 95 from:

    Warning ${LINENO} "Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set)."

    to these 2 lines:

    export OTHER_TEST_FLAGS="-RegisterForSystemEvents"
    RunTestsForApplication "${TEST_HOST}" "${TEST_BUNDLE_PATH}"

    as described in Xcode 4: Running Application Tests From The Command Line in iOS (and if you want to be able to run command-line builds with tests on any local developer environments, like xcodebuild -target MyAppTests -sdk iphonesimulator, this is the step to do on each of those as well)

  9. Run a build again and look for errors. Assuming you have application tests for portions of the application using Core Data, you may see an error relating to the storage URL like “Error validating url for store”
  10. To fix the error above (or prepare for it, if the application doesn’t use Core Data yet but will eventually), you’ll need to change several of the configuration methods for Core Data to operate differently during command-line tests as described in iOS Unit Testing with Xcode 4 and Core Data; not covered here is how exactly to detect if you’re in a test, which I did by checking the WRAPPER_NAME environment variable like this (and I have local data being written inside the generated ./build directory):
    * Determine if application is running from a test inside an external command-line build by xcodebuild.
    - (BOOL) isExternalBuild
        NSString *wrapper = [[[NSProcessInfo processInfo]environment] valueForKey:@"WRAPPER_NAME"];
        return wrapper != nil && ([wrapper rangeOfString:@"octest"].location != NSNotFound);
    Returns the managed object model for the application.
    If the model doesn't already exist, it is created from the application's model.
    - (NSManagedObjectModel *)managedObjectModel
        if (__managedObjectModel != nil) {
            return __managedObjectModel;
        if([self isExternalBuild]) {
            NSArray *bundles = [NSArray arrayWithObject:[NSBundle bundleForClass:[self class]]];
            __managedObjectModel = [NSManagedObjectModel mergedModelFromBundles:bundles];
        } else {
            NSURL *modelURL = [[NSBundle mainBundle] URLForResource:@"MyApp" withExtension:@"momd"];
            __managedObjectModel = [[NSManagedObjectModel alloc] initWithContentsOfURL:modelURL];
        return __managedObjectModel;
    Returns the URL to the application's Documents directory.
    - (NSURL *)applicationDocumentsDirectory
        if([self isExternalBuild]) {
            NSString *dir = [NSString stringWithFormat:@"%@/build", [[NSFileManager defaultManager] currentDirectoryPath]];
            return [NSURL fileURLWithPath:dir];
        } else {
            return [[[NSFileManager defaultManager] URLsForDirectory:NSDocumentDirectory inDomains:NSUserDomainMask] lastObject];
  11. Run another build and look for errors. If you see a more generic error along the lines of “UISplitViewController is only supported when running under UIUserInterfaceIdiomPad” or messages like “Timed out trying to acquire capabilities data.” during the run, your build server may not have the necessary scheme metadata from Xcode.
  12. To fix the error above, first make sure you’ve checked the Shared checkbox in the Manage Schemes dialog of your project, which will place some additional metadata in the project directory that you can commit to source control (but make sure you don’t then ignore them!) as described in Managing Schemes
  13. If you still see the error the iOS simulator probably has not been launched on the build machine since the last restart; it must have run once to start the necessary services but also cannot be open when the test runs, so to handle this automatically you may want to add a simple execute shell targetat the beginning of your build like:
    open '/Developer/Platforms/iPhoneSimulator.platform/Developer/Applications/iPhone'
    sleep 3
    killall 'iPhone Simulator'
  14. Run another build and hopefully this one works! If not, post the additional steps required and I will add them to this checklist.
  15. Once everything is working perfectly in Jenkins you’ll want to confirm that you can still run both the application and any test targets successfully in the simulator or on a device directly through Xcode. The steps here should support this unless you’ve missed something.

, , , ,

  1. #1 by Mar on June 12, 2012 - 6:36 am

    After doing step 8, unit tests doesn’t run even from Xcode.
    I’ve got the following error:
    “Test host ‘${TEST_HOST}’ exited abnormally with code ${TEST_HOST_RESULT} (it may have crashed).”
    in RunPlatformUnitTests.include

  2. #2 by xem phim sex online on March 17, 2014 - 3:36 pm

    You really make it appear really easy together with your presentation however I in finding this matter to be actually one thing that I feel I would never understand. It kind of feels too complicated and extremely vast for me. I am looking forward for your next put up, I will attempt to get the cling of it!

(will not be published)