Top Tips for DevOps Friendly XCTest in iOS Pipeline

Shashikant Jagtap

Apple has introduced XCTest framework to create unit, integration, performance and UI tests for various Apple platforms. XCUITest is a sexy extension on top of XCTest which allow developers to write UI Tests for iOS and macOS apps. Since launched in WWDC 2015, XCUITest got a lot of attention and enhancement. It uses Swift language to create tests which makes it easy for iOS developers to add UI tests for the iOS applications using the same programming language. On another hand,  DevOps and Continuous Delivery practice enticing businesses to release new features as soon as it gets developed. As per the tradeoff, Unit and Integration tests are lightweight and faster. However, UI Tests are very slow brittle and unmaintainable. Many mobile testing pyramids are shown that we should do minimum UI Tests and cover the business logic lower level lightweight tests. However, UI Tests are irreplaceable and give much more value at a higher level if done smartly. A single UI Tests can beat thousands of unit tests in terms of confidence of releasing the app. In order to mix up DevOps and XCTests practices, we have to make some smart decisions on how to write XCTest without disturbing CI/CD pipelines. In this post, we will see some of the techniques for the DevOps friendly XCTests.

iOS CI/CD Pipelines

In order to enable frequent releases of iOS apps, it’s super important to set up and maintain CI/CD pipelines. One of the painful thing in the iOS development is to release the app. It involves many activities that can be error-prone boring, repetitive and time-consuming. The role of CI/CD pipelines is to automate all the boring tasks. The typical iOS release pipelines consist of following things

  • Check out the Source Code
  • Static Analysis of the Swift code ( e.g SwiftLint)
  • Compiling and Building  iOS app
  • Running different kinds of tests e.g unit, Integration, API, UI, Performance, Security
  • Code Sign and Archive iOS app
  • Distribute iOS app to iTunes Connect or Third-Party testing services
  • Create tags or release on Source Control e.g Github Releases
  • Last but not least, Notify people about the new build with release notes

There might be more things in the pipeline but some of the very common things listed above. It’s obvious that CI/CD pipelines won’t be complete without automated tests.

Top Ten Tips for DevOps Friendly XCTest

The automated tests provides value and confidence to release an iOS app without any fear. Now, we will see how to write XCTest smartly to speed up pipelines without compromising quality of iOS app.

1. More Unit and Integration Tests

Its always good idea to cover all the production code with unit tests so that Continuous Integration can detect bugs earlier in the lifecycle. The unit tests are usually lightning fast and trivial to write. With XCTest, we can cover most of the code by writing unit tests. In some cases, we have to write integration tests where unit testing is not enough to cover the business logic. Benefit of having lightweight and faster tests produces faster builds which provides early feedback. Apple has great documentation on writing unit tests with XCTest framework here

2. Contract Tests for Backend API

Its very common practice that iOS apps uses backend or server side API’s. It’s very important to ensure that APIs used by iOS apps are following the consumer-driven contracts. Otherwise, there are chances that they can break the iOS app completely. It’s essential to have contract and communication between the teams providing API to the iOS team in order to avoid breaking an iOS app. This initiates need of contract testing, which ensures communication between API provider and client without brittle integration tests. With XCTest framework, we can draw the contracts and automate network tests. We can also use Pact testing tools in Swift to perform contract tests. There is library on Github for contract testing with Pact using Swift here

3.Performance Test of API

The speed of an iOS app is a key to success in the current competitive market. XCTest framework has measure blocks which can be applied to any code inside the test method to check the performance of the code. However, backend APIs should be performance/load tested separately by the teams responsible for building those APIs. You can read this article for detailed guide on performance testing with XCTest. Adding performance tests for iOS release pipeline makes it more stable and we can detect performance issues earlier before we upload app to iTunes Connect. However, performance tests are very very slow and can be only run when needed e.g before release or as part of the nightly builds.

4. Accessibility Identifiers for UI Tests

Apple has invested heavily in Accessibility and it has been baked in UIKit framework. An Accessibility makes iOS apps usable for everyone including people having disabilities, it also help in UI Test using XCTest. You can read adding accessibility for UI Tests here  Adding an accessibility for UI elements makes UI tests much faster which reduces the entire build time.

5. Limit UI Tests

XCTest framework can be extended to User Interface tests a.k.a XCUITests. We can add as many UI tests as we want but we should keep in mind that UI tests are slow, brittle and time consuming. We should try to test most of the business logic in the integration and API level and keep critical user journeys for UI Tests. It’s possible to run XCUITests in the headless mode in any macOS server machine, still they are slow and need to keep the minimum.

6. Smart use of  Devices Tests

When XCTest framework is used for testing iOS apps, it launch simulators to execute tests. However, since Xcode 9, XCTest and XCUITests can run in headless mode when run from the command line. Any macOS server can run XCTest but its not enough to run tests inside simulator, we need them to run on real devices as well. Before releasing an iOS app, we should take feedback from real devices. On Continuous Integration server, you can attach some devices if you have self-hosted CI Or you can use cloud testing services like SauceLabs, Perfecto or AWS Device Farm.

7.Avoid Third-Party Test tools

Apple’s XCTest tool or other developer tools are enough to perform majority of quality assurance checks. There are are some other third-party Swift libraries like Quick for doing the spec level unit testing which adds unnecessary dependencies in the iOS app.  There are also third-party UI testing tools like Appium or Calabash which allows to write tests in other languages like Java, Ruby etc. These tools are more dangerous to build collaboration between developers and QA engineers and hard to plug into CI/CD pipelines as it requires unnecessary technologies to be installed and additional brittle configuration to be done on CI server. Using native tools  like XCTest makes CI/CD pipeline much easier to configure maintain. If your project is native (Swift) keep it native, don’t introduce any dependencies unless needed. Using irrelevant technologies increases build time and complexities of the projects e.g introducing Java or Ruby inside iOS projects doesn’t work out well.

8. Separate Build Configuration

An iOS app has Debug and Release build configurations by default. It’s always a good idea to create another build configuration for testing purpose which we can call Test or something like that. Having that flexibility we free up debug configuration for internal users settings. We can still add some test-related configuration for the Test configuration. This also reduce the risk of leaking the test code to the production app. Using different build configuration allows us to add test specific code which makes test execution much faster and covers most of the business logic with tests.

9. Don’t Build Twice

Compiling and building an iOS app takes lot of time. Some teams uses different schemes to build app for unit and UI testing. This adds unnecessary build time in the process of Continuous Integration. Its good idea to use xcodebuild features like build-for-testing  and test-without-building  as mentioned here. We can also use xctestrun  file for distributed CI build machine in order to save the build time.

10. Use right CI Servers

In an iOS development, its very important to use right Continuous Integration server to run all our XCTests that we wrote for our iOS app. It could be self-hosted or cloud-based CI system. There are some reasons mentioned here that CI can fail completely if not used or selected properly.

Conclusion

Fortunately, XCTest framework is a multipurpose tool, it can be used for the unit, testing, integration testing, API testing and performance testing. This reduces the burden of adding third-party tools to cover different types of testing. In short, XCTest is here to rule them all and making iOS test more and more DevOps friendly. What are the techniques you use for making XCTest DevOps or CI friendly? Please waive in the comments below