Automation DevOps Featured iOSDev

Hands-on XCUITest Features with Xcode 9

At WWDC 2017, there was a great session on What’s New in Testing which was mainly about new features of XCTest and XCUITest frameworks.  The team working on developer tools at Apple has made huge improvements in the area of  UI Testing with XCUITest and Continuous Integration with Xcode Server. In this post, we will explore all the new features with practical examples in Xcode 9 and command line. There is Github repo  Xcode9-XCTest created as part of this exploration.  You can reference this post to that GitHub repository to try out the things by your own.

There are a lot of new things announced related to testing for Apple platforms especially iOS and macOS. Some of them are as below.

  • XCUISiriService
  • Localisation Testing
  • Async Testing
  • UI Test Performance Improvement
  • Activities, Attachments and Screenshots
  • Multi-App Testing
  • xcodebuild: Headless Testing
  • xcodebuild: Parallel Testing
  • Inbuilt Xcode Server

There is a lot of enhancement in testing processes including Siri integration, Waiting in XCTest, Core Simulators in xcodebuild and much more. Let’s dive into this one by one.

XCUISiriService

Using XCUISiriService, we can now interact with Siri interface. We can control Siri by passing voice recognition text from our XCUITests and Siri will act accordingly. Let’s imagine,  we want to open an Apple News app using XCUISiriService, we can do that from our test.

I wrote a detailed blog about Controlling Siri from XCTest  using XCUISiriService a few months ago when it’s announced with Xcode 8.3 also created demo project on Github XCUISiriServiceDemo but API syntax has changed a bit but examples are still valid.

Practical Example

Open the Xcode9-XCTest project in Xcode 9 and run the testXCUISiriService() test from Xcode9_XCTestUITests.swift file. You can see the Siri will open Apple News app.

Localisation

With Xcode 9, we can use Xcode scheme to run the test in different Language and Region. When we edit the scheme, we see the Test options to run tests using specific languages and Regions. We can easily test our app with different language and regions by changing the scheme settings.

Async Testing

There are many situations where we need to wait till things to happen to carry on our test execution like opening document, waiting for a response from server etc but it’s most common in UI testing scenarios. As of now, XCTest handles async testing by creating expectations and test will wait till the expectation fulfilled. The XCTest timeout will result in test failure. The expectations are tightly coupled with XCTestCase.

Now, XCTest has new class XCTWaiter which allows us to define expectations explicitly which is decoupled from XCTestCase. It has initialiser as public API and then different waiting conditions covered. XCTWaiter waits for a group of expectations to be fulfilled. We can now define the expectations like this:

The list of the newly added classes and sub-classes  are as follows:

There is one more handy API that we can use to wait for existence of XCUIElement.

Practical Example

Open the Xcode9-XCTest project in Xcode 9 and run the testXCWaiterfeatures() test from Xcode9_XCTestUITests.swift file. I have already shared detailed article a few months ago on Asynchronous iOS Testing in Swift with XCWaiter also there is demo project on GitHub here 

 

UI Testing Performance Improvements

Removal of Snapshot

As we are aware of the fact that, XCUIApplication is a proxy app that interacts with the main app which is used to launch, terminate and query the main application. Apple was using snapshot tool to communicate between  XCUI app and main App which was causing performance issues in terms of time and memory. Now snapshot has been removed and replaced with remote query and query analysis technique to improve the performance of the XCUItests. We might have faster execution of the tests.

First Match API

Apple also introduced First Match API to speed up a response to the query. First Match API will exit early as soon as it finds the first XCUIElement rather than querying everything. We can use firstMatch on any XCUIElement

Now the query would have faster magnitude and no memory spike.

Match First Vs Match All

Activities, Attachments and Screenshots

There are big improvements in the area of organising tests for readable reports, taking screenshots of XCUIElements and attaching rich data to test reports. There are three different concepts introduced this year which are

  • Activity: Grouping of actions by giving meaningful name
  • Screenshot: Taking Screenshot of fullscreen or specific XCUIElement
  • Attachments: Attaching rich data to XCTest reports.

Activities

UI Tests are usually long running and a lot of actions happening there e.g tapping buttons, swiping etc. As of now,  XCTest reports show all the actions in the test reports which is not particularly readable. Activities are the way to organise those actions into the group by giving a meaningful name so XCTest result will use that activity name in the result to make it more readable. You can read more about activities on Apple official documentations here
We can sprinkle activities on any set of actions e.g launch app in clean state

Now that, we have defined an activity to launch an app in the clean state. When we run the test then in the test report we will see “Given I have launched the app in clean state”  which is more readable. We can still access underlying actions by expanding the activity.

ScreenShots

Apple also announced the new API to take a screenshot of full screen as well specific XCUIElements. There is a new protocol XCUIScreenshotProviding to provide a screenshot of current UI state and there are two new classes XCUIScreen and XCUIScreenshot to capture a screenshot of the app or UIElement state. We can take a screenshot using following code snippet.

Attachment

An Attachment can be used to attach rich data to test reports. It may be data from test, improved triage additional log, post processing workflow. The data may be in the form of raw binary data, string, property list, code object, files or images. We can attach the screenshots to activities using an attachment. We can add an attachment to activity like this:

Practical Example

Open the Xcode9-XCTest project in Xcode 9 and run the testActivitiesScreenShotsAttachments() test from Xcode9_XCTestUITests.swift file. You can see the test reports in Xcode are taking activity name rather than actions. In another test, we have attached screenshots to the activity.

Multi App Testing

Previously we can test only one application which is target application for the XCUITest target. There was no way to interact with other apps like Settings or News from XCUITest. XCUIApplication() is used to launch, terminate and query an application under test. It can only access target application under test and doesn’t test other applications. There were always need to access other applications like Settings or App extensions.

With Xcode 9, we can test multiple application using XCUIApplication. It has three main changes, now XCUIApplication()

  •  has initialiser which takes bundleIdentifier so that we can pass bundleID of our app
  •  has new activate() method to activate app from background
  •  has States to monitor changes in the application. States has cases like runningBackground , runningForeground etc etc

We can test two applications in one XCTest using below code

The activate() method automatically wait for another app to become active but there might be situations where we want to manually wait for the app to be activated with can done by using predicates and XCUIApplication states.

As we can see that, we can interact with any app within Simulator or device as long as we know bundle identifier. This is a huge improvement in UI testing of iOS apps.

Practical Example

Open the Xcode9-XCTest project in Xcode 9 and run the testMultipleApps() test from Xcode9_XCTestUITests.swift file. We are interacting with main application and settings app.

Note: Currently test is failing while launching settings up but the simulator is launching settings app regardless.

xcodebuild : Headless Testing

There are some huge improvements in the xcodebuild command line tool which is used for analysing, building, testing and archiving an iOS application. xcodebuild will use core simulator to run tests so we don’t see simulators running when running our tests from command line. It means we won’t see simulators on CI servers. It will be totally headless.

Practical Example

Clone or download  Xcode9-XCTest project and run xcodebuild command. One thing to note that, there is no simulator launching while tests are running.

xcodebuild: Parallel Testing

xcodebuild support parallel device testing. It means we can run the test in multiple simulator and devices as long as devices are provisioned also we can run a test on devices which are wirelessly connected to the server. We just need to pass multiple destinations to xcodebuild.

Practical Example

Same as above Clone or download  Xcode9-XCTest project and run xcodebuild command by passing additional destinations options.

Inbuilt Xcode Server

Xcode Server is a continuous integration system provided by Apple.  As of now, Xcode Server needs macOS Server application to run Xcode bots. Now,  Xcode Server no longer needs macOS Server application. It’s inbuilt in Xcode and can be accessed from Xcode Preference. Xcode Server has improved provisioning, uses core simulator (headless) for running test, parallel testing multiple devices, Localisation control.

Practical Example

I have shared a detailed article about how to setup Xcode Server using Xcode 9 which is Xcode 9 + Xcode Server = Comprehensive iOS Continuous Integration. Feel free to navigate to that article to learn more about Xcode Server setup from scratch.

Watch below how to create Xcode bot for our demo project  Xcode9-XCTest

Conclusion

New features in XCTest framework like parallel testing, automatic provisioning, multi-app testing will definitely be useful for all the iOS teams across the world. Apple is investing time on improving testing and continuous integration practices. Hope it will continue in recent years.

  • Arif M

    Very informative blog. Thanks for sharing.

  • Ashley

    Great tutorial… Can we save attachements without sctivities ?

  • Ashley

    Great tutorial.. can we save attachment without activity