Before diving into this article, make sure you take a look at the intro article on iOS CI Olympics game here.
In the introductory post on iOS CI Olympics, we have selected the few player and few games for this competition. We will be evaluating the player one by one on the basis of their performance in the each game. In this article, the player is CircleCI, a CI servers used by popular tech companies.
In the iOS Continuous Integration Server Olympics competition, the third candidate selected is CircleCI, the continuous integration service that is being used by the most popular tech firms like Facebook, Spotify, Percolate etc. The CircleCI and TravisCI have most of the things common but some features are unique to the each platform. The little bit information of CircleCI is as follows
Headquarters: San Francisco, California
Description: CircleCI provides software teams with the confidence to build, test, and deploy across numerous platforms.
Founders: Paul Biggar, Allen Rohner
Categories: Web Development, Developer Tools, PaaS, Software, Test and Measurement
SignUp & Project Setup
The CircleCI signup process was very straightforward, I have used my Github credentials to login into the CircleCI portal. The project XCFit is based on macOS platform so, I was looking for the option to set it with macOS. However, CircleCI doesn’t offer the free plan for the macOS platform. As per the documentation here, they provide support to the open-source project in the seed plan. In order to get the seed plan, I have to email email@example.com and ask for it. I email and waited for few days. After some days, a guy replied that they are looking into it and come back. I am still waiting for their reply. At that stage, I got two options, either use CircleCI’s free Linux container and just build the swift package on it, not the iOS project. OR enrol into the free trial of 15 days to try out the feature of CircleCI. I have decided to use free trial for this game and then swift to free container after 15 days. It’s shame that I can not build the open-source iOS project on CircleCI but I can do it with TravisCI, BuddyBuild and Bitrise.
Selecting trial version doesn’t give me satisfaction as CircleCI became over smart and detected XCFit project as Linux project rather than macOS, however other Xcode projects are detected as macOS projects.
Thankfully, I managed to setup this as macOS project in the next screen. Like TravisCI, I have added circle.yml in the project with following content that I wanted to execute in the macOS platform.
- gem install fastlane --no-ri --no-rdoc --no-document
- brew install carthage
- cd XCFit-Example && carthage bootstrap --platform iOS
- swift --version
- cd XCFit-Example && fastlane scan -s XCFit-Example
As I have already triggered a build in the Linux, it keeps running the builds in the Linux container. I have to choose something from the below screen. I selected this configuration to get the build working
Finally, I managed to run the build on macOS platform. The build started running, what happens to build we will continue in the default build configuration section below.
Medal: CircleCI has earned the Bronze medal for the project setup game
Default Build Configuration
The default build configuration for the CircleCI project can be set up from the user interface or by adding the project configuration in the source code. In case of the XCFit project, I already added the config mentioned above to be used in the build. I was expecting the build will start with the specified configuration but build started with default CircleCI configuration, CircleCI used the Xcode project XCFit.xcproj and Started Installing CocoaPods which I never wanted. It failed to the building project with the below error.
I spent some time to disable the CircleCI’s default awesomeness and use my configuration for builds. Finally, I found the way to override the configuration by setting the XCODE_PROJECT and XCODE_SCHEME environmental variables.
I also need to disable the default configuration of the CircleCI from the user interface.
I need to add the script echo "Ignore CircleCI defaults" in the Dependency Commands and Test Commands section. After all this configuration I managed to execute the project with my own configuration.
Medal: CircleCI has earned the Silver medal for the default build configuration game.
Build Execution Process
As soon as, we push the commit to Github repo, CircleCI will trigger the build. We can configure the builds for branches or pull requests. The CircleCI boots brand new VM, check out the source code and execute the configure scripts. The build execution process is very detailed and informative. We can each step in the separate section, which is handy. The Build logs are very clear and have different tabs to navigate.
I am heavily impressed with the build execution process of the CIrcleCI.
Medal: CircleCI has earned the Gold medal for the build execution game
Build Artefacts and Test Reporting
CircleCI has the separate tab to display the build artefacts and report the status of the tests. The build artefacts generated from the tests are looks like this:
In case of the XCFit, I have used Fastlane to execute the unit and UI tests. The test execution and reporting were awesome.
Medal: CircleCI has earned the Gold medal for Build Artefacts and Test reporting game
Real Device Testing
CircleCI runs the tests inside the macOS virtual machines that get killed once the build is finished. There is no possibility to add real device testing support with CircleCI. However, we can write custom scripts to trigger the test on real devices provided by third-party providers like AWS Device farm or Saucelabs
Medal: CircleCI has the earned Silver medal for Real Device Testing game.
CircleCI has various options to customise the build configuration from the user interface. CircleCI has options to environmental variables, input test and dependency commands, notifications, deployments and third- permissions settings.
Medal: CircleCI has earned a Gold medal for Build Customisation game.
Parallelisation & Concurrent Builds
CircleCI has various plans to provide the parallelism and concurrency. CircleCI seems to be very powerful in providing the parallel and concurrent builds
Medal: CircleCI has earned the Gold medal for Parallelisation and Concurrent Builds game.
CircleCI has provided me with the option to configure the workflow but the XCFit project isn’t big enough to set that up. The workflows feature is amazing. It seems to be launched recently as per this blog post so might be little young.
Medal: CircleCI has earned the Gold medal for Build Pipelining game.
Infrastructure As a Code
CircleCI is designed to work as an Infrastructure as Code. The whole build pipeline has to be coded in the circle.yml file. However, CircleCI provides the mixture of the configuration that can be done with UI as well as config file which can be confusing.
Medal: CircleCI has earned the Gold medal for Infrastructure as Code game.
Directory Caching: CocoaPods & Carthage
CocoaPods and Carthage are the two most used dependency manager for iOS projects. Both dependency managers check out the dependency source code and build the frameworks in Pods and Carthage directory. This is a very time-consuming process to do for every build. We need to cache those directories to speed up the build execution. At the time of writing this post, CircleCI support for caching the CocoaPods or Carthage directories. We have used the settings in the config file to cache the Carthage directory to speed up the builds. CircleCI has detailed documentation here as well we have already configured the caching for the Carthage directory.
Medal: CircleCI has earned Gold medal forDirectory Caching game.
Code Signing & Distribution
CircleCI has inbuilt support for the code signing of the iOS apps. We just have to upload the certificate and provisioning profiles to the CircleCI interface.
Medal: The areas that I haven’t fully tested with XCFit projects, I will be giving the Silver medals by default untested but available features.
Third Party Integrations
CircleCI has official support for the most used third-party provided that are being used in iOS development. There is support for JIRA, Webhooks for Slack, Hipchat. There is also deployment integration with AWS and Heroku.
Medal: Circle has earned the Gold medal for Third Party Integrations game.
CircleCI boots fresh VM for the new build. The process of booting, checking out the source code and installing pre-defined software can take a long time. The actual build will start once all the setup has been some. CircleCI builds are bit slower than I expected.
Medal: CircleCI has earned the Silver medal for speed.
CircleCI has an option to store all the secret in the form of an environmental variable that can be accessed from the settings page of the build.
We can inject the access tokens, encryption keys and all sort of sensitive information here. CircleCI will store then and won’t print as part of build execution. However, CircleCI seems to be not hiding all the information which can be guessed.
Medal: CircleCI has earned the Gold medal for Secret Store game.
Xcode & Apple Developer Tool Upgrades
CircleCI seems to be right in time to update the Xcode version and other tools. As per this blog, CircleCI has updated to Xcode 9 very next day. However, there are some of the discussions in the Google search shows that people requesting them to upgrade.
Medal: CircleCI has earned the Silver medal for fast Upgrades of Apple developer tools.
Open-Source Project Support
CircleCI provides a free Linux container for the open-source community which isn’t useful for building a iOS apps. There is no free support for running macOS builds. There is seed plan to build some open-source project but I am not sure what criteria to be selected for the seed plan. CircleCI sucks in supporting open-source projects.
Update: CircleCI provided with support for XCFit as seed plan so upgrading to Silver medal.
Medal: CircleCI has earned the Silver medal for Open-Source Project Support.
CircleCI configures the build by himself but it’s not an ideal approach. The user has to script all the build requirements in the config file. There is a little bit of learning curve when we get going but once learned its fun. The user needs to learn the scripting CircleCI features and integration to the third party but it gives all the power to the user.
Medal: CircleCI has earned the Silver medal for Learning Curve.
Documentation & Support
CircleCI has amazing user documentation which guides the user from getting started to configuring complex builds for multiple languages. However, the documentation for building iOS projects seems very limited. As of now, I found only one page to support for iOS apps here. CircleCI has a page for the discussions and forums which seems very informative.
Medal: CircleCI has earned the Gold medal for Documentation and Support.
CircleCI also has clean and intuitive UI that helps users to set up the projects easily. However, using CircleCI with Safari is very painful, links don’t respond in time, scrolling the pages moves entirely commands in the dependency commands and test commands. User profiles are replaced by question mark or strange emoji.
Update: I found that this issues with my browser cache. Upgrading to Gold.
Medal: CircleCI has earned the Gold medal for Clean UI/UX.
CircleCI offers four plans for running macOS builds. Starting with seed plan and going to Mobile focussed plan which allows us to run 12 concurrent builds.
Medal: CircleCI deserves the Gold medal for the price game.
CircleCI: Pros and Cons
As of now, we have seen the features of the CircleCI in brief, now the time is to decide what’s good and not good with CircleCI. There are certainly some great features that CircleCI offers but there are some limitations as well.
The benefits of using CircleCI are as follows
- CircleCI has very strong parallelisation
- CircleCI is very easy once setup it was done
- CircleCI has very clean build and test reporting mechanism.
- User can get SSH support for debugging the builds
- CircleCI has strong support for third-party service like AWS, JIRA
- Easy configurations for the notifications
- CircleCI has huge community
There are some pitfalls usingCircleCI which are as follows:
- CircleCI lacks support to build open-source projects
- CircleCI builds are bit slow
- Steep learning curve for the new users to set up projects on CircleCI
XCFit: Build Details
CircleCI Medal Summary
|Total GOLD||12G, 7S, 1B|
|Sign Up & Project Setup||🥉|
|Default Build Configuration||🥈|
|Build Execution ||🥇|
|Build Artefacts & Test reporting||🥇|
|Real Device Testing||🥈|
|Parallelisation & Concurrent Builds||🥇|
|Infrastructure As a Code||🥇|
|Directory Caching: CocoaPods & Carthage||🥇|
|Code Signing & Distribution||🥈|
|Third Party Integrations||🥇|
|Xcode & Apple Developer Tools Upgrades||🥈|
|Open-Source Project Support||🥈|
|Documentation & Support||🥇|
CircleCI medal counts looks impressive. In the rest of the Olympics competition, we will see how other players will perform. We will publish the final medal table at the end of the competition. In the next post, we will announce the results of the Oympics.. Continue Reading