Q&A: Automation Guild Conference XCUITest Session

Shashikant Jagtap

Recently, I spoke at the online conference called Automation Guild 2018 on the topic of Getting Started with XCUITest for iOS Apps. It was awesome conference organised by Joe Colantonio. You can find the details of my talk here. It was fully online conference having pre-recorded session and 30 minutes of live Q&A with the speaker. In this post, I will share my Q&A session with my detailed answers in written with extra details.

Why Write-Up?

My full session is 73 minutes including 45 minutes of the hands-on demo of the XCUITest. On the conference day, there was a Q&A session for 30 minutes where I answered some questions. The full Q&A session has been recorded by Joe, and he will be publishing on the conference website. You might be wondering why I am writing about the Q&A session if you got recorded video available to view. I watched my Q&A sessions and realised that for some questions, I made assumptions,  I didn’t understand properly, I baffled a couple of times or I wasn’t sure answer I provided satisfy the attendee who asked the question from users who asked in real time. As Automation Guild being an online conference, its bit tricky to get clarification on the question. I feel that there are some questions I could have answered better if its face to face conversation. I want to convey my answers in details to everyone who attended the conference as well as who couldn’t make the conference.

If you already got the ticket then you can view my full session and Q&A of on the automation guild site but here is the quick preview of my talk.

[embedyt] http://www.youtube.com/watch?v=3suPb75rxDg[/embedyt]

Now, we will cover the questions asked during live Q&A session.

Q1: Tell us something about yourself?

Since April 2016,  I am involved with DevOps, CI/CD and Test Automation activities for iOS apps. I have been enjoying  XCUITest framework. I blog about everything I do so you can always visit my blog a.k.a XCBlog to know what I am doing currently here. My blogs also get published on Medium, DZone, LinkedIn and Apple News. On top of that, I wrote some opensource tools  RubyGems (BDDfire and XCFit), Ansible roles on Ansible Galaxy (iOS-Dev, Xcode Server), Swift Packages on Cocoapods (XCFit BDD framework for iOS apps), Docker Images on DockerHub  (Cucumber-BDD, Appium, Gatling). Feel free to check out my Github Shashikant86 for more projects. I am the organiser of few meetups in London including London Test Automation in DevOps, London Jenkins Users Meetup and newly formed London Server Side Swift meetup. My full bio on Automation Guild conference page here.

Q2: What is the best to work with localising strings so that test runs in different languages?

It’s very common these days to have an app in different languages. If your main application supports localized strings for different languages then you probably have localization bundle with all the values for different languages. As XCUITest uses accessibility Identifiers to identify elements uniquely so it’s important to have unique accessibility identifiers regardless of translated strings so that we can test our apps in different languages even though the strings are different. Your tests should check the elements (labels, buttons etc) not the data. You can still add localised strings as accessibility labels which can be read by assistive technology like voiceover. If you really really want to test the localised strings then you can reuse the bundle in XCUITest and assert the strings are populated correctly.

In XCUITest, Apple has provided the couple of launch arguments like AppleLanguages  and AppleLocale  that can be used to force tests to run in different languages and locales. You can read my blog post on localization and internationalization testing using XCUITest here. With Xcode 9, you can update scheme and pass language and locales, you can read latest features here. This GIF shows how to edit Xcode scheme.

Hope, this answers this question. Let me know if you need more details or something I missed.

Q3: What is the good approach to testing on simulators and real devices?

iOS Simulator just simulates the iOS software environment. It does not attempt to emulate hardware so you won’t get the environment that users will be using with simulators. It’s always worth to run tests on real devices if possible but if you are running tests to get quick feedback or using cloud CI services then its ok to use Simulators. Good idea to run your pull requests on simulators and merge to the main branch on real devices.

It’s rare but it can happen that test runs ok on Simulator but fails on the real device or get slow or throws other problems. Its good idea to get feedback on your tests on real devices. If you want to know the difference between real devices, emulators and simulators, read this awesome post.

Q4: While setting up the target for iOS, there is the cross-platform tab, what are the platforms supported?

To be honest, I never clicked that tab but its more for cross-platform games and the Xcode template screenshot looks like this

In recorded Q&A, I said I am not sure but something like React native or something but there is nothing like React here.

Q5: Do you know if it is possible to run tests in parallel on multiple devices or simulator?

Yes. That’s now natively supported by Apple. Previously there was a limitation that you can run only one instance of Simulator but now we can run multiple simulators at the same time. At that time Facebook created tool xctool and LinkedIn created Bluepill to run XCUITests in parallel as a workaround. Now, Apple officially and natively supports parallel XCUITest both on Simulators and Real Devices. As a result, Apple broke Facebook’s  xctool and Bluepill still works but not required now.

You can run tests in parallel on Continuous Integration server using xcodebuild  tool. We just have to pass multiple destinations (Simulator or Device Id) to xcodebuild. I have covered this as part of my hands-on XCUITest blog post here with Demo Github repository Xcode9-XCTest. Clone or download  Xcode9-XCTest project and run the xcodebuild command by passing additional destinations options.

If you use Xcode Server as CI server then it has inbuilt option to run tests in parallel and you can attach many devices to the server as well. Read more about latest Xcode Server features here

In case of other CI servers, you can use xcodebuild command line tool mentioned above.

 Q6: How do you handle iOS apps that have iPhone and iPad designs?

This is an interesting question and glad that Raffel brought that up. If you are designing XCUITest for both iPhone and iPad then you have to architect those properly. Again accessibility Identifiers can help to identify XCUIElements uniquely regardless of the screen size so adding accessibility Identifiers is key to success.However, there are other aspects that you need to consider as well such as modal dialogue, popovers, action sheet where you need to add extra code to handle this logic. By the way, you can differentiate iPhone and iPad using UIDevice API from Apple.

Also, it’s a good idea to create separate Xcode Schemes to run tests for better reporting on multiple devices.

Q7: What are the third party device labs that support XCUITest?

This is another great question bought up by Jason. There are various vendors who provide the ability to run XCUITest in their device cloud. Let’s cover few of them

  • Bitbar

You can connect to Android and iOS Devices on the Bitbar public cloud.

  • AWS Device Farm

There is AWS Device Farm where you can upload the app and run tests. There is great documentation covering setup guide here.

  • Perfecto

Perfecto also support XCUITest in Parallel on the real device more on setup on Perfecto blog here. Perfecto is also one of the sponsors of Automation Guild conference.

  • Microsoft App Center

The Microsoft App Center or Xamarine test cloud also supports XCUITest on the real device. Microsoft has great documentation here also they wrote SDK AppCenter-Test- XCUITest-extension which is available on Github.

Remember, you have to have the packaged application ready to upload to cloud before you start running tests. If you want to setup Continuous Integration server to connect to Device lab, there is additional setup required.

Q8: Your tests are using UIElements like buttons and other assets that you defined.  Can you define the element from the code project, saving you the effort of setting up code repository?

This question is brought up by Carlson. I still didn’t understand the question correctly but with XCUITest your source code of main app and XCUITest code has to be in the same repository. You can’t separate the development code and test code. You need to get have all these assets in the main app then XCUITest access those assets using accessibility API using the proxy app. It would be great to get more insight on the question. The code repository I set up is on GitHub AutomationGuild-QuizApp

Q9: When XCUITest records a test, can they implement page object that you created? Can they create one itself?

When XCUITest is being recorded by native recorder then it won’t implement any pattern. The tests will be recorded as plain actions within same test method. However, I wrote a RubyGem called XCFit which can install Page Object pattern templates for you. Just run

This will create new target templated in Xcode and while creating new targets, you will see XCFit templates for Page objects and other patterns.

I would rather prefer Protocol Oriented approach than Page Object Pattern. Basically, these pattern like Page Object etc have come from Selenium community which doesn’t make that sense for XCUITest and Swift world. You can read more about XCFit and how to use each pattern here

Q10: Do you use those three test strategies depending on the types of the tests Or you always use same styles?

This question is brought up by Mark. I assume types of tests means unit, API, service, UI etc. Basically, XCUITest framework is designed only for UI level test so we cannot perform other types of testing. XCUITest runs in the separate process, where we can not access data, classes from application code. The main app communicates to XCUITest runner app only using launch arguments and Launch Environments. So it’s totally black box framework and designed only for UI level testing. We can still achieve performance testing with XCUITest but it is much smoother to do at the unit level.

In my demo, I used Page/Screen object pattern and Protocol Oriented Pattern. I always use protocol oriented pattern as Swift is protocol oriented language and it’s much easier to use extensions and enums to maintain the tests. Page Object Pattern is more likely for Java’ish kind of people.

Q11: Are there such things as flaky tests in XCUITest? If yes why do they happen? and how often?

Yes. There is hardly any place where flaky tests don’t exist. There are multiple reasons for flakiness in XCUITest but the most common reason is lack of accessibility Identifiers. There are lots of applications developed before Swift didn’t have accessibility inbuilt so XCUIElement locators are brittle causing brittle and flaky tests. Another common reason is state of your application and Simulator or real device where your tests are running. You must have to set your application and simulator in the desired state before running tests. There were few issues in XCUITest frameworks itself which Apple has fixed/fixing. However, if you have clean application is architected well, then there is less chance of flakiness. In my previous role, we experienced a lot of flakiness as it was legacy app full of tech-debt, no accessibility. We put some hacks like re-running flaky tests that I mentioned here, writing defencing code etc but if your app architecture is solid there won’t be any flakiness.

Q12: Any Good Blogs, Books references to learn more about XCUITest?

XCUITest framework has been launched on WWDC June 2015, it’s still young to write or have a book on it. However it’s on my TODO list, I am waiting for WWDC 2018 to see more updates and hoping to start working on it. Apple’s documentation & WWDC sessions are best places to learn more about XCUITest than reading any books. I will link some good resources to follow

WWDC Sessions

Apple Documentation

Other Blogs

There are some popular posts worth reading

You can always google it and see what comes first.

Thanks

Again huge thanks to Joe Colantonio for organising and Manoj Kumar and Richard Bradshaw for making me aware and recommending me to speak at this conference. See you next year at Automation Guild 2019. Hope, this post answers all questions asked about XCUITest session of Automation Guild conference. Let me know if you need more details or something I missed in the comments below.