iOS CI Olympics : Competition of Cloud iOS Continuous Integration Servers

Shashikant Jagtap

Continuous Integration a.k.a CI for iOS apps is one of my favourite topics. Continuous Integration is essential for every iOS project now as the number of developers growing in the iOS teams.  As of now, I have deeply experienced with self-hosted CI servers like Xcode Server, TeamCity, and Jenkins for iOS projects. However, there are still an array of cloud-based CI services available in the market which are becoming very popular in the recent days. There are a lot of articles and tweets going on for those cloud-based CI services. I am keen to try those services and get some exposure by using them with my favourite personal project XCFit.  In order to do this, I have decided to play an Olympics game involving the most of the cloud-based iOS Continuous Integration services candidates.

Purpose

The most of the services taking part in this competition that I haven’t used before apart from TravisCI. It would be the good opportunity for me to make myself familiar with those CI services and compare them using my personal project XCFit. The purpose of this Olympics competition is to share my experiences and first impressions on those cloud-based iOS CI services. Please note that purpose of this competition is not to recommend particular service that you should use for your iOS project. 

The whole purpose of this competition is to try and share the first impressions of these cloud-based iOS CI services.

Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of anyone else in this world. This evaluation is only performed against  XCFit project and it may vary from project to project. 

Olympic Players

The player involved in this Olympics games are the most used cloud-based CI servers which supports iOS and macOS platforms. The current list has these players

XCFit: The Game

This Olympic competition will be played against the project XCFit, the library we are going to use to try out the features of CI services mentioned above. This is one of my favourite pet project XCFit which is the small library that allows us to write a XCUITests in Behaviour Driven Development a.k.a BDD fashion. This project is interesting candidate for CI server test as it consists of

  • Swift Framework distributed with CocoaPods and Carthage
  • Swift Package that can be distributed with Swift Package Manager
  • An Example app using XCFit framework built with Carthage
  • Ruby package distributed using RubyGems to download Xcode templates

As there is a lot involved in this project so it would be interesting to see how all those cloud-based CI servers treat this project to run the builds on respective platform.

Olympic Athletics

The Olympic athletics, we will be using for this competition are

  • Sign Up & Project Setup
  • Default Build Configuration
  • Build Execution
  • Build Artefacts & Test reporting
  • Real Device Testing
  • Build Customisation
  • Parallelisation & Concurrent Builds
  • Build Pipelining
  • Infrastructure As a Code
  • Directory Caching: CocoaPods & Carthage
  • Code Signing & Distribution
  • Third Party Integrations
  • Speed
  • Secret Store
  • Xcode & Apple Developer Tools Upgrades
  • Open-Source Project Support
  • Learning Curve
  • Documentation & Support
  • Clean UI/UX
  • Price

The players will be evaluated on basis of these games. Each of the athletics will earn a medal that would be gold, silver or bronze. At the end, there will be pros and cons section where we will discuss the outcome of this evaluation. This will give the brief idea about the functionality of the cloud-based iOS CI servers and their features related to iOS development.

20 Olympic Medals

Each of the game mentioned above will earn a medal. The medals we are going offer are Gold, Silver and Bronze.

  • Gold Medal

The player will get a gold medal for the criteria if it is strong to satisfy user and met requirements clearly as well as exceptionally.

  • Silver Medal

The player will get the Silver medal for the specific criteria if it is performed alright and satisfy user up to the certain extent.

  • Bronze Medal

The player will get a Bronze medal if it’s met the minimum requirements but doesn’t satisfy user that much. This is the least medal that every athletics will get for the performance.

Each player will get 20 medals and the winner will be decided on the basis of the number of Gold medals. There is no good or bad CI services but the winner will be announced based on how each cloud-based CI servers will perform against XCFit project only. There might be different winners for different projects.

5 Magic Medals: At the end

There are 20 different games in the Olympics and each player will get a medal for each game. This means every player will get 20 medals for sure but there will be additional 5 magic medal that will be offered to the player while the evaluation is going on and announced once all players are evaluated. The 5 medals will be based on following criteria

  • Social Media support during evaluation
  • Correcting my observations during evaluation
  • Accepting facts and respecting competitors
  • Showing courage to improve services
  • User Feedback on Twitter & Public API

Series of Blog Posts

We will have series of blog post on the cloud-based iOS CI servers Olympics from now on, we will link from here

  1. iOS CI Olympics-Player One: BuddyBuild
  2. iOS CI Olympics-Player Two: Bitrise
  3. iOS CI Olympics-Player Three: TravisCI
  4. iOS CI Olympics-Player Four: Nevercode 
  5. iOS CI Olympics-Player Five: CircleCI 
  6. Announcing Winner of the 2017 Cloud iOS CI Olympics

Winner

The winner will be decided on the basis of a maximum number of gold medals. There will be total 25 medals 20 from Olympic games and 5 magic medals. The medal table will always show who is leading but the winner will be announced after 5 magic medals are announced at the end of the tournament. So Stay tuned!

In the next post, we will see the performance of the BuddyBuild. Continue Reading