Mar 30 2013
Mobile Test Automation
Mobile testing is a real challenge when it comes to automation because of variety of the platforms, versions and interfaces. Let’s say, iOS Apps, We got iPhones, iPads and different versions of them. Android devices are even more complex. Mobile test automation is a emerging topic in recent days and selecting automation libraries takes lot of research and investigation of the variety of test automation tools. In this tutorial, I will explain how I set up Calabash, Cucumber test automation framework on Mac OSX to test Vim iOS application.
Calabash: Mobile Test Automation Library
Calabash is a mobile test automation technology which currently supports iOS and Android Apps. Calabash is maintained by company called ‘LessPainful‘. Github repositories are maintained by Karl Krukow [@karlkrukow].
I got introduced with Calabash while attending Tech Talk by Moncef Belyamani at AOL. He has worked on various AOL mobile apps automation by using combination of Calabash and Cucumber. Watch his talk at Mobile Testing Summit last year on ‘Calabash: Tips and Tricks‘. By reading his blog post ‘iOS Automated Testing with Calabash, Cucumber and Ruby’, I have decided to dive directly in to Calabash without spending time on evaluating other tools. Calabash seems like awesome option for mobile test automation.
Vim iOS Application
There are number of open-source iOS Apps available here but I have selected Vim iOS application. It’s because Vim is different than other application because of it’s interface and usage. It doesn’t have Buttons, Fields, Placeholders or links. I thought, it would be challenging to test text editor like Vim on mobile devices.
You can find source code on GitHub here. You need to setup Vim app as a Xcode project. It’s simple, from your Xcode, Goto File->Open->Vim[your_project].
Get Started with Calabash
I asume that, you have already setup development environment on your Mac by using Moncef’s blog. Basically, you need to install following tools and services.
- Xcode with Command line Tools
- Various versions iOS Simulators. [ It comes with Xcode]
- Homebrew [Package Management ]
- RVM and Ruby gems
I have already setup all these things on my Mac OSX. Again I would strongly recommend Moncef’s blog to do this.
Download Vim iOS Source-Code
Once you done with all the necessary setup, you need source code for your application. You can clone Github repository directly into your Mac
$ git clone https://github.com/applidium/Vim
Now, it would be an awesome idea to separate RVM for Calabash project from other Ruby application.
$ cd Vim/ $ rvm --rvmrc --create use 1.9.3@calabash
Now, It’s time to install Calabash-Cucumber gems into our Vim directory & setup Calabash framework. You need to run following commands in sequence
$ gem install calabash-cucumber $ calabash-ios setup $ calabash-ios gen
These commands will setup Calabash Framework and create sample Cucumber feature file into your project. Here is screenshot.
We need to hit return after this step to continue installation.
You can setup Xcode project manually by following step by step instructions on Moncef’s blog but I think, you can still use ‘Fast Track‘ which will do most of the setup thing automatically and then you can perform checks if setup is right or wrong.
In order to do that, you need to run ‘cucumber’ command which will fail complaining about Xcode setup but it will perform most of the task mentioned in the ‘Manual Setup’
Now, here are some important checks, you need to perform on Xcode in order to confirm status of your Xcode setup
Ensure Scheme [-cal]
First thing, you need to ensure that, new scheme is automatically generated. This duplicated project is called ‘Vim-cal’. You will see screen something like this:
Check Build Setting: Product Name
In your Xcode, Go to Build Settings and Search for ” Product name”. Make sure result id ‘Vim-cal’
Check Calabash Framework in the App
You can simply ‘ls’ into your project and see if ‘Calabash-Framework’ directory is available.
Shashi-MacBook-Pro:Vim user$ ls Contents README_bindos.txt Users runtime Contents.info README_dos.txt Vim.info runtime.info Filelist README_extra.txt Vim.xcodeproj Makefile README_mac.txt Xxd.info README.txt README_ole.txt calabash.framework src README.txt.info README_os2.txt configure src.info README_ami.txt README_os390.txt csdpmi4b.zip uninstal.txt README_ami.txt.info README_src.txt farsi vimdir.info README_amibin.txt README_srcdos.txt features vimtutor.bat README_amibin.txt.info README_unix.txt libs vimtutor.com README_amisrc.txt README_vms.txt nsis README_amisrc.txt.info README_w32s.txt pixmaps
Ensure Link Binary with CFNetwork.framework.
Go to ‘Build Phase’ and expand ‘Link Binary with Libraries’ You have to make sure you got ‘ CFNetwork.framework’ library.
Ensure Other Linker Flag
Go to ‘Build Seting’ and Search for ‘ Other Linker’. You should see flag is pointing to ‘Calabash framework’
Ensure Accessibility Turned ON.
In order to perform some functions on iOS app you need to turn, Accessibility ON feature ON. In order to do that, you need to RUN Vim-Cal App. Once emulator is launched, Check the ‘Output’ by clicking on the ‘View’ in the top right corner.
Go to ‘Settings->General->Accessibility” and toggle ‘ Accessibility Inspector ‘ON’. You will see something like this.
Now that, we have done with all the setup. It’s time to execute Cucumber to see the auto generated feature is passed.
Now, you will see auto generated feature is passed and screenshot is saved in the project directory.
Add Another Feature
Now , we will add another feature into out Vim project.
$ sudo vim visual_mode.feature Feature: Test visual mode of the Vim editor In order to use Vim As a Vim user I want to enter on the visual mode Scenario: Test visual mode Given I am on the Welcome Screen When I swipe left And I wait for 2.3 seconds Then I don't see the "VIM - Vi IMproved"
This feature is using predefined Calabash Steps so that we don’t need to write implementation code. Now we will run Cucumber to see all steps passed. Watch the video demo below.
In this post, we have seen testing look and feel of Vim using predefined Calabash Steps. In the future post, we will see
- Using Calabash Custom Steps & Ruby Api’s to test Vim iOS app
- Testing all features of the Vim like insert mode, copy, syntax etc etc.
I know, it will be challenge, you can join me to contribute to this project. Please feel free to fork Github and Contribute.
GitHub : Calabash-Cucumber-Vim
Calabash is a awesome automation framework for testing iOS and Android apps by using Cucumber. Calabash helps QA, Developers to write robust automated test suits to test mobile App. Again, we look forward, Calabash support for the Behat.
Happy Mobile Testing !!