At WWDC 2017, there was a great session on What’s New in Testing which was mainly about new features of XCTest and XCUITest frameworks. The team working on developer tools at Apple has made huge improvements in the area of UI Testing with XCUITest and Continuous Integration with Xcode Server. In this post, we will explore all the new features with practical examples in Xcode 9 and command line. There is Github repo Xcode9-XCTest created as part of this exploration. You can reference this post to that GitHub repository to try out the things by your own.
There are a lot of new things announced related to testing for Apple platforms especially iOS and macOS. Some of them are as below.
- XCUISiriService
- Localisation Testing
- Async Testing
- UI Test Performance Improvement
- Activities, Attachments and Screenshots
- Multi-App Testing
- xcodebuild: Headless Testing
- xcodebuild: Parallel Testing
- Inbuilt Xcode Server
There is a lot of enhancement in testing processes including Siri integration, Waiting in XCTest, Core Simulators in xcodebuild and much more. Let’s dive into this one by one.
XCUISiriService
Using XCUISiriService, we can now interact with Siri interface. We can control Siri by passing voice recognition text from our XCUITests and Siri will act accordingly. Let’s imagine, we want to open an Apple News app using XCUISiriService, we can do that from our test.
|
1 2 3 |
func testXCUISiriService() { XCUIDevice().siriService.activate(voiceRecognitionText: "Open News") } |
I wrote a detailed blog about Controlling Siri from XCTest using XCUISiriService a few months ago when it’s announced with Xcode 8.3 also created demo project on Github XCUISiriServiceDemo but API syntax has changed a bit but examples are still valid.
Practical Example
Open the Xcode9-XCTest project in Xcode 9 and run the testXCUISiriService() test from Xcode9_XCTestUITests.swift file. You can see the Siri will open Apple News app.
Localisation
With Xcode 9, we can use Xcode scheme to run the test in different Language and Region. When we edit the scheme, we see the Test options to run tests using specific languages and Regions. We can easily test our app with different language and regions by changing the scheme settings.
Async Testing
There are many situations where we need to wait till things to happen to carry on our test execution like opening document, waiting for a response from server etc but it’s most common in UI testing scenarios. As of now, XCTest handles async testing by creating expectations and test will wait till the expectation fulfilled. The XCTest timeout will result in test failure. The expectations are tightly coupled with XCTestCase.
Now, XCTest has new class XCTWaiter which allows us to define expectations explicitly which is decoupled from XCTestCase. It has initialiser as public API and then different waiting conditions covered. XCTWaiter waits for a group of expectations to be fulfilled. We can now define the expectations like this:
|
1 2 3 4 5 6 7 8 |
wait(for: [documentExpectation], timeout: 10) // OR XCTWaiter(delegate.self).wait(for: [documentExpectation], timeout: 10) // OR let result = XCTWaiter.wait(for: [documentExpectation], timeout: 10) if result ==.timeout { ... } |
The list of the newly added classes and sub-classes are as follows:
- XCTWaiter : Takes control over to the waiting strategy.
- XCTNSPredicateExpectation: Readable expectation while writing automate tests
- XCTKVOExpectation: Readable expectation while writing automate tests
- XCTDarwinNotificationExpectation: Another readable expectation for notifications.
There is one more handy API that we can use to wait for existence of XCUIElement.
|
1 |
XCUIElement.waitForExistance() |
Practical Example
Open the Xcode9-XCTest project in Xcode 9 and run the testXCWaiterfeatures() test from Xcode9_XCTestUITests.swift file. I have already shared detailed article a few months ago on Asynchronous iOS Testing in Swift with XCWaiter also there is demo project on GitHub here
UI Testing Performance Improvements
Removal of Snapshot
As we are aware of the fact that, XCUIApplication is a proxy app that interacts with the main app which is used to launch, terminate and query the main application. Apple was using snapshot tool to communicate between XCUI app and main App which was causing performance issues in terms of time and memory. Now snapshot has been removed and replaced with remote query and query analysis technique to improve the performance of the XCUItests. We might have faster execution of the tests.
First Match API
Apple also introduced First Match API to speed up a response to the query. First Match API will exit early as soon as it finds the first XCUIElement rather than querying everything. We can use firstMatch on any XCUIElement
|
1 |
let button = app.navigationBars.buttons["Done"].firstMatch |
Now the query would have faster magnitude and no memory spike.
Match First Vs Match All
|
1 2 3 4 5 |
app.buttons.firstMatch //Not Good idea app.buttons["Done"].firstMatch // Better app.navigationsBars.buttons["Done"].firstMatch //best |
Activities, Attachments and Screenshots
There are big improvements in the area of organising tests for readable reports, taking screenshots of XCUIElements and attaching rich data to test reports. There are three different concepts introduced this year which are
- Activity: Grouping of actions by giving meaningful name
- Screenshot: Taking Screenshot of fullscreen or specific XCUIElement
- Attachments: Attaching rich data to XCTest reports.
Activities
UI Tests are usually long running and a lot of actions happening there e.g tapping buttons, swiping etc. As of now, XCTest reports show all the actions in the test reports which is not particularly readable. Activities are the way to organise those actions into the group by giving a meaningful name so XCTest result will use that activity name in the result to make it more readable. You can read more about activities on Apple official documentations here
We can sprinkle activities on any set of actions e.g launch app in clean state
|
1 2 3 4 |
XCTContext.runActivity(named: "Given I have launched app in clean state") { _ in XCUIApplication().launch() XCUIApplication().launchArguments = ["-StartFromCleanState", "YES"] } |
Now that, we have defined an activity to launch an app in the clean state. When we run the test then in the test report we will see “Given I have launched the app in clean state” which is more readable. We can still access underlying actions by expanding the activity.
ScreenShots
Apple also announced the new API to take a screenshot of full screen as well specific XCUIElements. There is a new protocol XCUIScreenshotProviding to provide a screenshot of current UI state and there are two new classes XCUIScreen and XCUIScreenshot to capture a screenshot of the app or UIElement state. We can take a screenshot using following code snippet.
|
1 2 |
let screen = XCUIScreen.main let fullscreenshot = screen.screenshot() |
Attachment
An Attachment can be used to attach rich data to test reports. It may be data from test, improved triage additional log, post processing workflow. The data may be in the form of raw binary data, string, property list, code object, files or images. We can attach the screenshots to activities using an attachment. We can add an attachment to activity like this:
|
1 2 3 4 5 6 7 |
XCTContext.runActivity(named: "When I add attachment to activity") { (activity) in let screen = XCUIScreen.main let fullscreenshot = screen.screenshot() let fullScreenshotAttachment = XCTAttachment(screenshot: fullscreenshot) fullScreenshotAttachment.lifetime = .keepAlways activity.add(fullScreenshotAttachment) } |
Practical Example
Open the Xcode9-XCTest project in Xcode 9 and run the testActivitiesScreenShotsAttachments() test from Xcode9_XCTestUITests.swift file. You can see the test reports in Xcode are taking activity name rather than actions. In another test, we have attached screenshots to the activity.
Multi App Testing
Previously we can test only one application which is target application for the XCUITest target. There was no way to interact with other apps like Settings or News from XCUITest. XCUIApplication() is used to launch, terminate and query an application under test. It can only access target application under test and doesn’t test other applications. There were always need to access other applications like Settings or App extensions.
With Xcode 9, we can test multiple application using XCUIApplication. It has three main changes, now XCUIApplication()
- has initialiser which takes bundleIdentifier so that we can pass bundleID of our app
- has new activate() method to activate app from background
- has States to monitor changes in the application. States has cases like runningBackground , runningForeground etc etc
We can test two applications in one XCTest using below code
|
1 2 3 4 5 6 7 8 9 10 11 |
func testTwoApps() { let app1 = XCUIApplication(bundleIdentifier: "com.app1.xyz ") let app2 = XCUIApplication(bundleIdentifier: "com.app2.xyz") // Launch App1 app1.launch() // Launch and test App 2 app2.launch() app2.launchArguments = ["-StartFromCleanState", "YES"] // test App1 again by activating it from back ground app1.activate() } |
The activate() method automatically wait for another app to become active but there might be situations where we want to manually wait for the app to be activated with can done by using predicates and XCUIApplication states.
|
1 2 3 |
let app1ActiveExpectation = XCTNSPredicateExpectation(predicate: NSPredicate(format: "state == \(XCUIApplication.State.runingForeground.rawValue"), object: app1) wait(for: [app1ActiveExpectation], timeout: 10) |
As we can see that, we can interact with any app within Simulator or device as long as we know bundle identifier. This is a huge improvement in UI testing of iOS apps.
Practical Example
Open the Xcode9-XCTest project in Xcode 9 and run the testMultipleApps() test from Xcode9_XCTestUITests.swift file. We are interacting with main application and settings app.
Note: Currently test is failing while launching settings up but the simulator is launching settings app regardless.
xcodebuild : Headless Testing
There are some huge improvements in the xcodebuild command line tool which is used for analysing, building, testing and archiving an iOS application. xcodebuild will use core simulator to run tests so we don’t see simulators running when running our tests from command line. It means we won’t see simulators on CI servers. It will be totally headless.
Practical Example
Clone or download Xcode9-XCTest project and run xcodebuild command. One thing to note that, there is no simulator launching while tests are running.
|
1 2 3 |
$ git clone https://github.com/Shashikant86/Xcode9-XCTest $ cd Xcode9-XCTest $ xcodebuild -scheme "Xcode9-XCTest" -destination 'platform=iOS Simulator,name=iPhone 7 Plus,OS=11.0' build test CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO |
xcodebuild: Parallel Testing
xcodebuild support parallel device testing. It means we can run the test in multiple simulator and devices as long as devices are provisioned also we can run a test on devices which are wirelessly connected to the server. We just need to pass multiple destinations to xcodebuild.
Practical Example
Same as above Clone or download Xcode9-XCTest project and run xcodebuild command by passing additional destinations options.
|
1 |
$ xcodebuild -scheme "Xcode9-XCTest" -destination 'platform=iOS Simulator,name=iPhone 7 Plus,OS=11.0'-destination 'platform=iOS Simulator,name=iPhone 7 Plus,OS=10.3' build test CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO |
Inbuilt Xcode Server
Xcode Server is a continuous integration system provided by Apple. As of now, Xcode Server needs macOS Server application to run Xcode bots. Now, Xcode Server no longer needs macOS Server application. It’s inbuilt in Xcode and can be accessed from Xcode Preference. Xcode Server has improved provisioning, uses core simulator (headless) for running test, parallel testing multiple devices, Localisation control.
Practical Example
I have shared a detailed article about how to setup Xcode Server using Xcode 9 which is Xcode 9 + Xcode Server = Comprehensive iOS Continuous Integration. Feel free to navigate to that article to learn more about Xcode Server setup from scratch.
Watch below how to create Xcode bot for our demo project Xcode9-XCTest
Conclusion
New features in XCTest framework like parallel testing, automatic provisioning, multi-app testing will definitely be useful for all the iOS teams across the world. Apple is investing time on improving testing and continuous integration practices. Hope it will continue in recent years.