WWDC 2016 : Continuous Integration with Xcode Server and XCTest updates

Shashikant Jagtap

WWDC 2016

The Apple Worldwide Developers Conference ( WWDC 2016 ) brings us lots of improvements in softwares although there are no hardware release this time. There are many software release including iOS10, macOS Seirra, Siri on Mac, Xcode 8 etc etc there is brief  summary of the what has been announced at WWDC 2016 is here on The Next Web Blog. I would recommend watching the session “Platform State of the Union” which is kind of keynote for the Developers where you can find lots of the announcement about development tools including

  • Xcode 8
  • Apple File System
  • Swift 3 : Open-Source Status
  • Swift Playground app
  • Code Signing Improvements

In this post we will cover whats new in terms of the Continuous Integration and Testing.

Xcode Server and XCTest

Apple has some related sessions at previous WWDC conference. Here is list of those related sessions including

Unfortunately there is only one session about the the Testing and Continuous Integration this year of WWDC. You can watch full session on WWDC video on “Advanced Testing and Continuous Integration” Or click on the image below

AdvanceTesting_CI

Let’s cover what has covered in this talk as part of this post including enhancements in Xcode, Xcode Server and xcodebuild command line tools.

Whats new in XCTest

Xcode 8 now has feature of gathering crashes in the test report itself.

Testing as four characters

As mentioned in the talk, there are four stages of the testing at this stage while testing out app.

  • XCTest – Test Frameworks
    Used to write Unit and UI tests
  • Xcode – Test harness:
    used to author test, execute tests and review reports
  • Xcode Server Continuous Integration:
    used to schedule bot runs, generate reports and send notifications
  • xcodebuild- Command Line Tool :
    Used by Xcode server to build and execute tests and for custom CI system.

four_stages

Zoltan then explained fundamentals  of ‘testing timelines’ with two concepts

  • Test hosting

Depending on the test it’s been hosed on app, In unit test test hosting is done inside the app while in the UI testing hosting is done at the Test Runner. Unit tests can have direct access to data and APIs and all test running in the same launch while UI tests have not direct access to data and APIs as it uses accessibility to interact with  element so needs every test launch app for every test case.

test_hosting

  • Test Observation

The test observation is details process of observing behaviour of the details. It can be done using XCTestObservationProtocol. This protocol has some call backs including

Bundle Will Start, Suit Will Start, Case Will Start, Case Did Finish, Case Did Fail, Bundle Will Finish

xctestobservations

Test Crash reporting

This is brand new feature in the Xcode which will report all the crashes as part of the test report. This has following

  • Improved test reporting with crash logs
  • It works for both unit and UI tests
  • Logs includes in the test reports
  • View crashes in the Xcode Debug Navigator

xcode_crash_gathering

What’s new in Xcode Server

Xcode Server is a continuous integration sever can be used to run Unit and UI tests. Xcode Sever has lots of improvements in the latest release which includes

  • Custom environment variable
  • Advance trigger editing
  • Issue tracking and blame
  • Upgrade integrations
  • Configurable integration user

Let’s explore in the each of the feature in brief.

Custom Environment Variable

This isn’t new but Xcode 8 will now allow us set environmental variable without creation of extra schemes.

Advanced Triggers

Xcode 8 will will now have two types of triggers, script triggers and email triggers. We will have now control over how to set email contains.

Issue Tracking and Blame

Xcode Server will send email to person who broke build. We cab now configure bots instead of changing code.

Upgrade Integration

Xcode Server now re-integrate the your project if it fails using same revision. It also prevent blaming for the broken build.

Configurable Integration User

Xcode Server will have now it’s own use’ called ‘Xcode Server” which give you full control with password as any normal macOS user. Setting new user seems fairly easy in the Xcode Server

login_as_user

Some of the best practices using the Xcode Server user

  • Dedicate a new user
  • Avoid using administratiove accounts
  • Stay logged in with Fast User Switching
  • Disable Screen Lock
  • Customise need such as Simulators, Networks and advanced provisioning

Whats new in xcodebuild

xcodebuild is the command line tool to build, run and execute our application from command line. This is used in the Xcode server. Xcode 8 now has some improvements in the xcodebuild command line tools.

Xcodebuild now has two new testing options

  • –only-testing : Include Test suites
  • –skip-testing : Exclude Test suits

xcodebuild_new_options

Build For Testing

The xcodebuild now has ‘build-for-testing’ option, it takes workspace scheme and destination as usual but on top of that it will create ‘XCTESTRUN’ file.

xcbuild_fir_testing

 

Test without Building

The xcodebuid also has another option called ‘test-without-building’ where we don’t need to provide workspace instead we specify the ‘XCTESTRUN file which will inject that file and runs all the tests.

xcbuild_test_without_building

This feature can be highly useful for the distributed testing as we can created XCTESTRUN file at one machine and distributed to other test specific machine.

xcodebuild_build_for_testing

Those are the only updated from XCTest and Xcode Server from WWDC 2016. I will post more about performance, accessibility updates soon

Hope you enjoyed WWDC 2016 this year !!