WWDC19: Getting Started with Test Plan for XCTest

Apple’s WWDC conference is bringing some great features for developers. Xcode11 packed up with some amazing features too. Release notes of Xcode have all the latest enhancements. One of the great features announced this year was Test Plan for XCTest and Xcode UI tests. In this post, we will cover how details of how Test Plan feature will work with XCTest. There is a session on Testing in Xcode which will give more details and we will cover that in the follow-up post.

Xcode Test Plan

Before Xcode 11, the testing configuration was part of the Xcode schemes. If developers want to configure the different set of tests for different conditions, they have to either create a new scheme or edit the existing scheme for the specific testing need. The test configuration was tightly coupled with Xcode schemes which might end up creating multiple schemes for different testing needs.

This new feature in Xcode11, allows developers and QA engineers to configure the tests as per their needs. Test Plan manage 

  • Which tests to run as part of the build
  • How to run those tests e.g random or lexical
  • How to manage test artefacts e.g attachments and screenshots
  • How to run tests under different memory conditions and locations
  • How to use run time tools as part of your testing

With Xcode Test Plan, you can configure your test suits independent of Xcode schemes. That’s huge enhancements in the testing technologies which can extend your testing to the next level. Now, let’s see the Xcode Test plan in action.

Scenario

Let’s assume that we are creating a test plan for functional testing with the following requirements.

  • The functional tests should have smoke and regression suite.
  • Smoke tests need to be run in the London(UK) while the entire regression suite will run in San Fransisco(USA).
  • Smoke tests need to be run in alphabetical order while regression tests need to run randomly to catch dependencies issues between tests.

We will create this test plan for our XCTest for different Apple platform apps.

Enable Test Plan in Xcode

Xcode11 allows us to define these requirements in the test plan. Xcode allows you to create a test plan from your current scheme configuration. In Xcode, when you edit the scheme and navigate to the test action then Xcode will give you the option to create a test plan. You can create a Test Plan with the following option

  • Blank Test Plan: Gives you a blank template and define your own plan 
  • Test Plan from Scheme: Take the scheme configuration and define the Test Plan
  • Choose Test Plan: if you already created a test plan, then you can select that plan from the scheme. 

In order to get started, let’s create a test plan from the scheme configuration. 

You need to name the test plan and attach to the right test target. While the test plan is created, they are attached to the scheme. However, the scheme can override any test plan created late using the command line flag to xcodebuild  tool. Once the test plan is created, you can add them to the specific target. Let’s add this to the UI tests target and name them.

You will see the new file created in the target with functional.xctestplan  extension. This is known as the test plan. You can edit the test plan as per your need. You can create multiple test plan in the same target depending on the testing needs. At this point, we have enabled the test plan for our UI test target. 

Test Plan File

The test plan file is a simple JSON like configuration file which holds information on how to configure your test to run against any scheme. There are various options available in the test plan to configure the tests. The basic test plan has the following two key elements. 

  • Configurations
  • Test Targets
  • Settings

Each configuration in a Test Plan has a set of test attached to it with different settings. As per the requirements of our test plan we probably need two configurations one for smoke tests  and other for the regression  tests.

Now open the   functional.xctestplan  file in Xcode and rename the configuration1  with smoke-uk  and click on the + button at the bottom of the plan to add another configuration regression-usa 

Now that, we have two configurations in our test plan and our test plan already has the UITest target attached. Let’s add some settings for each configuration. When smoke-UK  configuration is selected, we can override the location to London also make other changes to the settings. Similarly, edit the regression-usa configuration. The test plan will look like this for regression requirements.

The setting available in the test plan are

  • Arguments

This allows us to override the launch arguments and Environments for the tests.

  • Localisation

This setting allows us to configure the tests on a different location, locale and language. This is a great feature for internationalization testing.

  • UI Testing/Attachment/Test Execution/Code Coverage

These options are specific to UI testing and test artefacts. You can decide to keep the artefacts or remove as soon as possible. Also, determine the order of test execution and code coverage for your test.

  • Xcode Code Diagnostic tools

There are multiple Xcode code diagnostic tools like address sanitizer, thread sanitizer available to detect the runtimes issues. You can also enable other tools that detect the memory leaks etc.

Creating Test Plan from Scratch

If you don’t want to create a Test Plan from the Xcode scheme, you can always create a new test plan from the Xcode->Product->Test Plan option.

You should be able to create Test Plan from there and attach it to some test target by adding the tests to the newly created Test Plan. In the example below, we have created a new test plan and attaching the tests to the new test plan.

Creating the Test Plan From Source

As mentioned earlier, the Xcode Test Plan is just a configuration file where we define all the configurations, we have discussed so far. In our scenario, we have created the functional.xctestplan  and source code for the test plan looks like this:

You can define the configuration and save it with the .xctestplan  extension in the Xcode11.

Executing the Test Plan

Now that, we understand the process of creating the Test Plan in Xcode. Let’s see how to execute the test plans and get the reports in the Xcode. You can run the test using the Xcode ->Product -> Test for your scheme as usual which executes the attached test plan for the scheme. In our case, it will execute both smoke and regression test configured in the functional test plan.

There are some command line options added to the xcodebuild  to list and run the tests plans. You can execute the following command.

This command will list all the Test Plans for the current scheme. You can also execute the test by passing the command line option for the XCTestPlan project.

This will execute the test using the configured test plan.

Reports of Test Plan

Once the tests have been executed with a specific test plan, you can see the nice reports generated in the Xcode. You can also see the test report in the test navigator section of the Xcode. The reports generated per configuration.

In the Xcode, test navigator, you can filter the test reports by Test plans. Xcode Test Plan reports can be configured on Xcode Server for better reporting.

Source Code of this demo available on Github: XCTestPlan

Xcode Test Plan: Use cases

You might be wondering where Xcode Test Plan can be used and how it’s different from scheme configuration. Some of the uses cases where Xcode Test Plan make sense.

  • Differentiate types of testing e.g Smoke Tests, Regression Tests as we have done in this demo.
  • Testing apps in different languages, locale and location e.g if you want to test your app in multiple languages you can create different configurations.
  • Setup heavy tests like performance, load and separate them from the normal testing. You can configure the tests with code diagnostics tools or other types of non-functional tests.

There might be many more scenarios where the Xcode test Plans can help to extend your testing workflow.

Conclusion

With the Xcode Test Plan, XCTest can be extended to the next level by the ability to control how to configure, execute your tests. Xcode Test Plans give better control for Apple platform testing. Again there is a section on Testing in Xcode which give more insight on Xcode Test Plan. Stay Tuned for follow up article after that session. Enjoy WWDC for now.