Posts Tagged testing

Code coverage for iOS testing with OCUnit

I recently posted steps for setting up iOS unit testing with Jenkins, but working on a new project realized I wanted to take this a step further and start seeing code coverage metrics. The configuration has evolved quite a bit but fortunately there’s a great Code Coverage with Xcode 4.2 guide from Infinite Loop that covers this and seems to work with both Xcode 4.3 and 4.4 in my testing. Be sure to check the included links, related posts, and followup in comments for troubleshooting various errors that creep up across different Xcode versions. Once tests have been run against instrumented code you can view the coverage output graphically in CoverStory on a Mac. Finally, if you want to convert this output to Cobertura-style for display in Jenkins using gcovr you’ll want to configure your Jenkins build as described in iOS Code Coverage, Cobertura and Jenkins. Good stuff!

, ,

1 Comment

Automated iOS Jenkins builds with application tests and Core Data

I recently started some iOS development with a new team and had the chance to set up a server for continuous integration builds. Since our teams are already familiar with Jenkins that seemed like the logical place to start, and the Xcode plugin available works reasonably well out-of-the-box for simply building the application. However, running tests – especially what Apple calls application tests (anything accessing a bootstrapped NSApplication through the simulator) using Core Data – took a little bit of tweaking. Here’s what was needed as of Xcode 4.2.

  1. Install the latest version of Jenkins, either on a Mac with Xcode installed or another server that has such a Mac available as a slave for any iOS builds
  2. Install a plugin for your source control system of choice if not already present (I’m using Git with the default Git plugin - considering the great Git support added in Xcode 4 with Git repos enabled by default, if you’ve been looking for an excuse to switch this is it!)
  3. Install the Xcode plugin
  4. Create a build following the plugin build instructions for a basic release build of the application, and especially pay attention to the keychain notes (I went ahead and entered the login password of the build account so that Jenkins could unlock it itself)
  5. Run a build and verify that everything is successful up to this point, and if not, troubleshoot
  6. Edit the build and add a separate build target for testing following the plugin test instructions, specifying iphonesimulator as the SDK (you can enter a full absolute path like the example if you wish, but simply iphonesimulator should find the latest), and make sure it points to your project’s test target instead of the primary build target (e.g. MyAppTests instead of MyApp)
  7. Run a build and look for errors. Assuming you have one or more application tests expecting to run in the iOS simulator it will probably have skipped all the tests with an error like ”Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).”
  8. To fix the error above (or prepare for it, if application tests don’t exist yet but will eventually), on the Jenkins server (or slave if running the iOS build on a slave) edit:
    /Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/RunPlatformUnitTests (if Xcode <= 4.2)
    /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/RunPlatformUnitTests (if Xcode >= 4.3)

    and change line 95 from:

    Warning ${LINENO} "Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set)."

    to these 2 lines:

    export OTHER_TEST_FLAGS="-RegisterForSystemEvents"
    RunTestsForApplication "${TEST_HOST}" "${TEST_BUNDLE_PATH}"

    as described in Xcode 4: Running Application Tests From The Command Line in iOS (and if you want to be able to run command-line builds with tests on any local developer environments, like xcodebuild -target MyAppTests -sdk iphonesimulator, this is the step to do on each of those as well)

  9. Run a build again and look for errors. Assuming you have application tests for portions of the application using Core Data, you may see an error relating to the storage URL like “Error validating url for store”
  10. To fix the error above (or prepare for it, if the application doesn’t use Core Data yet but will eventually), you’ll need to change several of the configuration methods for Core Data to operate differently during command-line tests as described in iOS Unit Testing with Xcode 4 and Core Data; not covered here is how exactly to detect if you’re in a test, which I did by checking the WRAPPER_NAME environment variable like this (and I have local data being written inside the generated ./build directory):
    /**
    * Determine if application is running from a test inside an external command-line build by xcodebuild.
    */
    - (BOOL) isExternalBuild
    {
        NSString *wrapper = [[[NSProcessInfo processInfo]environment] valueForKey:@"WRAPPER_NAME"];
        return wrapper != nil && ([wrapper rangeOfString:@"octest"].location != NSNotFound);
    }
    
    /**
    Returns the managed object model for the application.
    If the model doesn't already exist, it is created from the application's model.
    */
    - (NSManagedObjectModel *)managedObjectModel
    {
        if (__managedObjectModel != nil) {
            return __managedObjectModel;
        }
    
        if([self isExternalBuild]) {
            NSArray *bundles = [NSArray arrayWithObject:[NSBundle bundleForClass:[self class]]];
            __managedObjectModel = [NSManagedObjectModel mergedModelFromBundles:bundles];
        } else {
            NSURL *modelURL = [[NSBundle mainBundle] URLForResource:@"MyApp" withExtension:@"momd"];
            __managedObjectModel = [[NSManagedObjectModel alloc] initWithContentsOfURL:modelURL];
        }
    
        return __managedObjectModel;
    }
    
    /**
    Returns the URL to the application's Documents directory.
    */
    - (NSURL *)applicationDocumentsDirectory
    {
        if([self isExternalBuild]) {
            NSString *dir = [NSString stringWithFormat:@"%@/build", [[NSFileManager defaultManager] currentDirectoryPath]];
            return [NSURL fileURLWithPath:dir];
        } else {
            return [[[NSFileManager defaultManager] URLsForDirectory:NSDocumentDirectory inDomains:NSUserDomainMask] lastObject];
        }
    }
    
    
  11. Run another build and look for errors. If you see a more generic error along the lines of “UISplitViewController is only supported when running under UIUserInterfaceIdiomPad” or messages like “Timed out trying to acquire capabilities data.” during the run, your build server may not have the necessary scheme metadata from Xcode.
  12. To fix the error above, first make sure you’ve checked the Shared checkbox in the Manage Schemes dialog of your project, which will place some additional metadata in the project directory that you can commit to source control (but make sure you don’t then ignore them!) as described in Managing Schemes
  13. If you still see the error the iOS simulator probably has not been launched on the build machine since the last restart; it must have run once to start the necessary services but also cannot be open when the test runs, so to handle this automatically you may want to add a simple execute shell targetat the beginning of your build like:
    open '/Developer/Platforms/iPhoneSimulator.platform/Developer/Applications/iPhone Simulator.app/'
    sleep 3
    killall 'iPhone Simulator'
  14. Run another build and hopefully this one works! If not, post the additional steps required and I will add them to this checklist.
  15. Once everything is working perfectly in Jenkins you’ll want to confirm that you can still run both the application and any test targets successfully in the simulator or on a device directly through Xcode. The steps here should support this unless you’ve missed something.

, , , ,

3 Comments

Google IO 2010 day 2

Plenty more fun on day 2! There was lots of general iPhone and iPad killing with Android 2.2 demos during the keynote, along with the announcement of Google TV. The keynote is up here.

More on GWT from the sessions:

Widgets
There is lots of client-side performance improvement to be had using LayoutPanels, UIBinder, and ClientBundle. Another great workaround, which does have some risk and complexity for “maximum performance”, is innerHtml. The new components in 2.1+ help abstract that out so as consumers we don’t have to dip into innerHtml directly as often to get the benefits. There are also some good organization and code clarity benefits to UIBinder, like declarative views and replacing the monolithic app css file with bite-sized package files where possible.

Testing
This session made me happy because the testing best practices at Google seem to align really well with what my team is already doing. They stressed heavily using MVP to create unit-testable code and then test in a JVM as much as possible, keeping views so simple that there is very little which can break. They also use Selenium/WebDriver for high-level integration tests that walk through the app as a sanity check. There was a little bit of discussion about GWT unit tests (the compiled ones), by for the most part everyone agreed that they are too slow for the value they provide so it’s an intermediary step for special cases where not a whole lot of testing occurs. Most importantly teams should just acknowledge up front that there are multiple scopes of GWT tests and try to use them all correctly. Another good idea out of this was developing UI test harnesses for complex UI views, especially anything browser-specific or with performance constraints, so that it’s possible to write “Selenium unit tests” that bring up just the harness page with the widget and hit them really hard in a tightly controlled scenario rather than in the larger application (even a mock application).

GWT in Production Applications
This was Ray Ryan’s (of the infamous 2009 MVP session) architecture talk this year. The overall theme around MVP was the same, with some focus on broader topics like supporting bookmarkable/crawlable pages. However it was interesting to learn that GWT core is finally going to bring in some version of GWTLog that implements the Java logging API. The main tool used for demoing these was Spring Roo with GWT which in a nutshell is “Grails for GWT” – really cool to run a few commands and auto-generate your whole project, JPA entities, service endpoints, etc. but not something I’ll use on an existing, well-established product. For brand new products or teams just learning GWT it could be cool, and for the community overall certainly lowers the barrier of entry to get a full-stack Java/GWT project running quickly. Finally, in addition to logging there’s some new support in 2.1 for manipulating headers and some SafeHtml utils for using innerHtml in a way that doesn’t introduce security holes.

, , ,

No Comments

TDD antipatterns

These TDD antipatterns are hilarious, and all too familar to anyone who’s done test driven development with a wide range of aptitudes on the team!

, ,

No Comments