The speed of an iOS app is a key to success in the current competitive market. User hates slow running and unresponsive apps so it became very important to check the performance of an iOS app before it goes to the end users.

Apple announced performance testing support to XCTest framework at WWDC 2014. XCTest frameworks can measure the performance of the block of code inside the unit or UI tests. In this post, we will see how to perform automated performance testing using XCTest framework.

 

Historical Performance Testing in iOS

Apple has various developer tools to check the performance of iOS apps. The most common are Instruments, Leaks, Profiler, Allocations etc. These tools can help to detect performance issues, memory issues and other bottlenecks. Traditionally, an engineer has to babysit and perform all these checks manually over the period of time. This approach is not acceptable in the modern iOS development as small code change can make an app unresponsive, unstable and unusable. We can’t manually check performance issues all the time. There has to be automated way to check the performance of an app on regular basis. Fortunately, There are few ways we can achieve automated performance testing of iOS apps

• Running Instruments check on periodic basis
• Automated Performance Tests using XCTest framework.

In this post, we will cover automated performance tests using XCTest framework as instrument checks on a regular basis aren’t appropriate as it still involves repeatable tasks.

Performance Testing with XCTest

XCTest framework has measure blocks which can be applied to any code inside the test method to check the performance of the code.

• The measure block executes the code multiple time depending on the number of invocations set. The default value is ten, it means the block will run ten times.
• At the end of the execution, it calculates the average value that can be used as a baseline for the future test execution. We can also set the baseline value manually.
• The performance test will fail if an average time goes higher than the baseline.

Applying Performace Test at UI Level

Selecting the candidates for performance test is the decision of the team but as per the XCTest documentation, performance tests can be applied to any XCTest regardless the level of the test. It means we can apply measure blocks to unit test as well as UI tests. As per the most online articles, performance testing is only applied to Unit Tests. We can also apply performance test for the XCUITest so that it can communicate to all the application endpoints. This will also give us the opportunity to explore which service/API is responding faster/slower. Applying performance tests to UI tests has its own pros and cons. XCUITests will cover different areas but the performance test might become incredibly slow. If we apply performance tests at the unit level, we can get quick results. When and where to apply performance tests are outside the scope of this article but for this demo, we will apply for XCUITests.

Performance Test with Measure Blocks

In order to see how it works in the practice, let’s create a demo app XCUITest-Performance with UI Test target. Just add a button to the main storyboard with accessibility identifier “Press” without any IBAction. At this stage, we will have a demo app with a button. In the template XCUITest, create a test for the performance test which has measure block and inside the block, we can launch the app and press on the button. The typical test looks like this:

We can run the test from Xcode and see that the test will run the code inside the block ten times. When the test run first time it will fail and prompt developer to set the appropriate baseline value for the future test execution. The test will pass/fail according to the result of the average value and baseline value.

Watch this in the GIF below

Now that, we have set the baseline value for our future performance tests. The performance tests will run as our normal unit tests all the time but the test will fail if average value increased above the baseline. It means we can automatically test the performance of our code along with unit test execution.

Granular Performance Test with Measure Matrics

In the above test, we have calculated average time from launch of the app till user press the button. however, we can go granular and set the point from where we have to start and stop the measure of performance. It means, we can apply performance test to the specific area of the test code within the unit test by using start/stop method. We can achieve this using the measure Metric method.

 

Let’s write performance test only when pressed the button. We can launch an app in the normal way and then apply measure Metric method to start and stop measuring the performance.  We can add the test like this:

Now that if we run this test, we can see that application is launched only once but we can calculate the performance by multiple invocations.

The measure metrics is a great way to optionally set up start and end point for performance measurement during the test execution.

Result Analysis of Performance Tests

The results of the performance tests are represented graphically in the Xcode. The typical performance test result will look like this

The dialogue reports average, baseline and Max STDDEV values. In the resulting dialogue, we can see that the performance test is 0.779% worse than the previous execution. Those results can be also seen in the console output if we are running the performance tests from continuous integration server.

Pros and Cons of XCTest as Performance Test Tool

XCTest is unit and UI level functional testing tool, however, it is extended to do performance testing of iOS modules. There are some pros and cons of using XCTest as performance testing tool.

Pros

There are certain benefits to developers for using XCTest for performance testing. Some of them are as follows

• XCTest framework is core Apple framework so there is no need to add third party dependencies for performance testing.
• The usual Unit Tests written using XCTest can be easily extended to performance tests by adding measure blocks around the code snippets. No need to learn an additional performance test tools like Gatling or JMeter  etc
• Easy to write performance test using Swift. Noe need to learn additional languages like Scala or Java etc
• Easily pluggable to any Continuous Integration server as they can be executed using xcodebuild or Fastlane scan.
• Ability to match the baseline using device types, so no need to worry about comparing the results with valid data sets.
• Ability to use Xcode IDE for performance testing.

Cons 

Along with some benefits, there are certain pitfalls using XCTest for performance testing. Some of them are as follows

• The performance test results generated by XCTest aren’t helpful for the non-technical person. XCTest performance test reports can not be exported to the readable format.
• The performance test results can not be shared or hosted easily unlike Gatling or other performance test tools.
• As the number of performance test increases the test suite become slower so the tests have to be run separately from the development workflow.
• XCTest can not simulate large number of users to check the performance of an iOS app
• XCTest can not alter the hardware device setting like CPU, Memory etc to cover complex behaviours for performance tests.

Tips for writing accurate  performance tests

In the Xcode, performance tests aren’t treated separately. The XCTest framework allows us to run performance test along with our unit or UI tests. It becomes programmers responsibility to make sure that the test itself is setup correctly and working as expected. There are few things we can keep in mind while setting up performance tests

• Know What you are Testing

Before diving into performance test, we need to understand what and how we are going to test the specific code for performance. It would be a good idea to identify the level at which performance tests can be executed, it might be unit level, integration level or UI level. We can execute the test at any level as long as the test using XCTest framework.

• Avoid Conflicts between invocations

The performance test executes multiple innovations of the same code block. We have to make sure that each invocation is atomic and independent from another without having high standard deviation. Anyway, Xcode will report an error if there are any deviations.

• Control the Test Data

The predefined test data has high chances of producing reliable test results than random data while running a performance test. The random data will produce flaky results which aren’t useful to anyone.

• Run Performance Tests Separately in different Scheme

As the number of the performance tests increases the time taken by test suites increases significantly. It would be a great idea to configure the separate scheme to run the performance tests only without affecting the normal flow of unit and UI tests.

• Run Performance Tests Continuously on Continuous Integration Server

As we have created a separate scheme for performance tests then it would be fairly easy to configure with any Continuous Integration server. It would be super easy if you are using Xcode Server. We can create separate CI job to run the performance tests on regular basis e.g daily, overnight  etc

Source Code for this Demo

The source code of the performance test discussed in this article is available on Github. You can clone the source code and try to run performance tests by yourself.

Github Repo: XCTest-Performance

Conclusion

Apple has extended core XCTest framework for performance testing, which can help us to detect the performance bottlenecks in our iOS apps as early as possible in the development cycle. It would be fairly easy for any iOS developer to get started with performance testing without the need to learn new language or framework. With latest Xcode, it becomes even smoother. Hope you will try this yourself and plug some performance tests into your iOS apps soon!