Asynchronous iOS Testing in Swift with XCWaiter

Apple recently announced Swift 3.1 development snapshot  and XCode 8.3 for the developers. There are couple of handy classes added to the XCTest framework to enable Asynchronous Testing for iOS and macOS applications. In this post, we will see how we can perform asynchronous testing using XCWaiter.

Swift 3.1-Dev

Newly added classes are available in Xcode 8.3 which is currently available for the download if you have Apple Developer Account. You can get it from the Downloads section of the developer account. Xcode 8.3 needs macOS version 10.12 and above. You can download compressed XIP file which is around 4.52 GB.  If you already have previous version of the Xcode then remove it or you can keep it but you have to switch between Xcode DEVLOPER_DIR. Once downloaded you can extract the file to install Xcode 8.3 beta and wait for installation of Xcode and command line tools. Once Xcode 8.3 is fully installed with all the command line tools, we can drag it into  /Applications path. Now, we have to switch to the new Xcode version by running following command

This will set new DEVELOPER_DIR and  we are ready to use Xcode 8.3. Make sure you are using correct toolchain using  xcrun --find swift  command which will shows current tool chain you are using.

Now, make sure you export toolchain and using correct version of Swift which is  Apple Swift version 3.1-dev  at the moment. You can easily do that by running following commands.

This will ensure that you are using Swift 3.1. Now we are good to try new features of XCTest Framework.

Current Waiting in XCUI Test

The XCTest framework allows developers to write unit and UI tests for iOS, macOS apps. Apple introduced Xcode UI testing in the WWDC 2015 which allow us to write UI tests within the Xcode. As part of the Xcode 8.3 release, Apple has aded couple of new class to the XCTest framework to support asynchronous testing. It means there are no handlers involved while waiting for the  XCUIElement  to appear. Previously, we had  waitForExpectations(timeout: handler:) method combined with  XCTestExpectation  to test asynchronous code which looks like this:

This piece of code will wait for 3 seconds to find the button and it will fail after 3 second if it doesn’t find element. We have passed  nil  to handler which invokes error once  timeout is reached. This cause test to fail as well error thrown is very generic and not often useful while debugging.  Luckily, we now have better control over error and handlers with XCWaiter.

XCTestWaiter

The XCTest Framework now has XCTWaiter class to wait for the element. The list of the newly added classes and sub-classes  are as follows:

XCWaiter class returns results of the expectations in the form of boolean. It returns  enum of four possible situations those are  .completed ,  .timedOut ,  .incorrectOrder  or  .invertedFulFilment . We can simply add extension to  XCUIElement  or add simple function which returns result of waiter function which returns one of the result from XCWaiter.

There is no callback block or completion handler. The helper method simply returns a boolean indicating if the element appeared or not. Let’s briefly go through each of the boolean results returned by  XCTWaiterResult  to understand what it is doing

Completed

This returns true when defined expectations are fulfilled or completed. It returns false when expectations are not met within specified timeout. Tests will likely to fail when expectations not met.

TimedOut

This will return true when expectations are not met within timeout. We can defined how to fail our test case by providing useful error message for debugging.

Incorrect Order

This tells us the expectations which are succeeded and which are still waiting to succeed. We know that what is taking time to diagnose the problem.

InvertedFulfilment

Not entirely sure how it works but as per docs If an expectation is set to have inverted behavior, then fulfilling it will have a similar effect that failing to fulfill a conventional expectation.

Readable Expectations

There are some new expectations added to the XCTest Framework which are XCTNSPredicateExpectationXCTKVOExpectation and XCTDarwinNotificationExpectation. I think, idea behind that is to make expectations more readable and customisable. Previously we have to wtite expectations with handlers. Now we can write expectation like this :

We can then pass this expectation to the  XCWaiter to get boolean result. You can find some examples in the demo repo here.

What’s Benefit of XCWaiter

Better Waiting Strategy

XCWaiter gives us boolean results from each expectations which allows us to define our XCTest with better control. This might help to deal with flakiness of the tests. XCWaiter results simply return whether element appeared or not.

Ability to define Multiple Expectation

We can defined more than one expectation and wait for the XCWaiter to return result for each of them. As mentioned above expectations can be written in more readable way.

Handling Timeout & Error

Previously XCtest used to fail if expectations are not fulfilled within the specified timeout and error was very generic. Now we have .timedOut  result from XCWaiter so that we can control test failure with proper error message.

Reduce Flakiness

XCWaiter gives us more control how we want to define test failure. This will reduce amount of the flaky tests. The XCWaiter results can also give us ability to fail or pass tests with proper error message

Highlight Performance Issues

XCWaiter gives us ability to define multiple expectations and each expectation tried to be fulfilled within timeout period. There is XCWaiter result .incorrectOrder  tells us how many expectations are fulfilled and how many still waiting to fulfilled. This gives us indication why those expectations are slow and might have some performance issues. We can diagnose the slowness to make test and application faster

GitHub Repo XCWaiter Demo

I have created GitHub repo Xcode83_Demo with sample iOS application to give the XCWaiter try and experiment XCWaiter features. Feel free to clone and give it a go .

Let me know if you have other comments on XCWaiter.