Before diving into this article, make sure you take a look at the intro article on iOS CI Olympics game here.
In the introductory post on iOS CI Olympics, we have selected the few player and few games for this competition. We will be evaluating the player one by one on the basis of their performance in the each game. The first player was BuddyBuild and we have seen its performance in the previous post. In this post, the player is Bitrise, another mostly heard iOS CI service.
Bitrise
In the iOS Continuous Service Olympics competition, the second candidate selected is Bitrise. There are many articles on medium and lot of good words on Twitter about the Bitrise. Like BuddyBuild, the Bitrise is also designed for the mobile continuous integration which supports iOS and Android platforms. Bitrise also got some good clients like ABB, Foursquare, FOX, TNT etc. The little of the background of Bitrise organisation
Company: Bitrise
Founded: 2014
Headquarters: Budapest
Description: Mobile continuous integration and delivery platform, with dozens of integration.
Founders: Barnabas Birmacher, Viktor Benei, Daniel Balla
Categories: Android, Developer Tools, iOS, Apps, Developer Platform
Website:https://www.bitrise.io/
In this post, I will share my first impressions of getting started with Bitrise for my favourite pet project XCFit using the iOS CI Olympics games. So Let’s get started with our games
SignUp & Project Setup
Bitrise was good in connecting to the Github repositories. Just need to select the repository that we want to build on Bitrise. I have selected XCFit repository to build on Bitrise.
Now that, we have selected the app that we want to build, the next step that Bitrise automatically do is scan the repository with specific GitHub branch. In case of XCFit, its master branch
Medal: Bitrise has earned the Gold medal for the project setup game
Default Build Configuration
Once, we select the repository and branch to build then Bitrise checkout the source code from the repository and find out the iOS apps and stats building the project using the default build configuration.
As mentioned earlier, XCFit has an example app inside XCFit-Example directory, that has Cartfile to build the project using Carthage. The default configuration of Bitrise was failing to find the Cartfile located inside the XCFit-Example directory. The build was failing using the default configuration.
It took a lot of time for me to figure out the Bitrise workflows and how Carthage is being used. In the end, I have deleted the Carthage workflow step and added the custom script to fix this issue. Apart from that, the Bitrise process was great.
Medal: Bitrise has earned Silver medal for the default build configuration game.
Build Execution Process
Bitrise starts to build quickly, in my case within few seconds after commit or as soon as possible. The build execution logs are shown very clearly and cleanly. The logs are shown in the xcpretty format which is readable and there was a button to download full logs. Bitrise also has a separate tab for the app artefacts and test reports. However, this doesn’t impress me, as Bitrise is just reusing xcpretty for displaying logs and creating reports. What will happen if xcpretty stops working in the future? I am not sure they have alternative plan to overcome this potential issue
Update: After having conversations with Bitrise in the comments below, It seems like Bitrise can have multiple reporting formats and we can switch between them. Upgrading from Bronze to Gold medal.
Medal: Bitrise has again earned the Gold medal for the build execution game
Build Artefacts and Test Reporting
After the build has finished, the entire logs are available for the download. There are two tabs in the build execution, one shows the entire logs and other shows the apps and artefacts. The test reports are generated using xcpretty HTML reporter format which isn’t that helpful. There are no the code coverage reports for the unit tests.
Medal: Bitrise has again earned the Silver medal for Build Artefacts and Test reporting game
Real Device Testing
Bitrise has support to execute iOS test using simulators but there isn’t any way to run the tests on real devices both for Android and iOS apps. Bitrise uses vagrant virtual machines so it would be difficult to add support for running UI tests on real devices.
Update: After having a conversation with Bitrise representative in the comment, it’s possible to configure tests on real devices for Android and trigger tests on third-party services like Saucelabs, AWS Device Farm. Upgrading from Bronze to Silver medal
Medal: Bitrise has again earned the Silver medal for the Real Device Testing game
Build Customisation
The default build configuration of Bitrise can be easily overridden using the workflows features of Bitrise. There are various workflows available to customise the build steps. There is also the possibility to add your own custom script just in case, we need something to do in between the workflow. The workflows can be re-arranged accordingly. The environmental variables are pretty common in the Continuous Integration system, Bitrise also has the ability to define the environmental variables with its values.
The great feature of Bitrise is that it allows us to code the entire workflow using bitrise.yml file. This file can be used on the local machine of the developer as well.
Medal: Bitrise has earned the Gold medal for Build Customisation game.
Parallelisation & Concurrent Builds
Bitrise has price plans per concurrencies so there might be the possibility to execute the concurrent build per plan. I am not sure what would be the maximum limit for the concurrent build. Bitrise dashboard doesn’t show any option to select the multiple simulators so I don’t think there is the possibility to parallelize the test execution as well.
Update: As per the comment in the post, Bitrise can go up to the maximum number of concurrencies as users pay more. Please read the comment below which explains the details which I couldn’t understand in the first place. I have upgraded Bitrise to Gold medal
Medal: Bitrise has again earned the Gold medal for the Parallelisation game
Build Pipelining
Bitrise has the concept of workflows which allows us to define the steps or phases within a particular build, however, the pipeline should allow us to execute or configure the builds in parallel or sequential manner. Using Bitrise workflows smartly we can achieve the pipelining of the iOS deployments but its limited and workflows cannot be run in parallel.
Medal: Bitrise has again earned the Silver medal for Build Pipelining game
Infrastructure As a Code
Bitrise is great in allowing the user to configure the workflows using UI as well as providing the command line interface to define the build steps. This allows the user to test the Bitrise workflows in the local machines before they push to the source control. Bitrise has also awesome documentation to get started with it.
Medal: Bitrise has earned the Gold medal for Infrastructure as Code game.
Directory Caching: CocoaPods & Carthage
CocoaPods and Carthage are the two most used dependency manager for iOS projects. Both dependency managers check out the dependency source code and build the frameworks in Pods and Carthage directory. This is a very time-consuming process to do for every build. We need to cache those directories to speed up the build execution. Bitrise supports the caching of both Carthage and Cocoapods but it requires configuration of workflows. The forum has the solution to cache the Cocoapods and Carthage directories to speed up the builds.
But there is no official parameter to cache the directories.
Update: As per the comment from Viktor below, Bitrise has the clean mechanism of caching the both Carthage and CocoaPods. Bitrise has added cache: pull and cache: push mechanism that applied to the default build. Read this discussion to know more here. Bitrise got Gold medal for this game
Medal: Bitrise has again earned the Silver medal for Directory Caching
Code Signing & Distribution
The code signing with Bitrise seems to matter of giving an access to Apple Developer Portal or upload the certificates to Bitrise. I think everything else like downloading provisioning profiles and code signing apps has been taken care by Bitrise However, code signing features haven’t been tested as part of the XCFit project.
Medal: The areas that I haven’t fully tested with XCFit projects, I will be giving the Silver medals by default to untested but available features
Third Party Integrations
Bitrise seems to have so many third-party integrations available in the workflow. There are various steps which cover the third party apps like SwiftLint, SonarQube, AWS etc.
Medal: Bitrise has earned the Gold medal for the third party integrations game.
Speed
Bitrise build usually starts and runs quickly. The executing the build was fast enough but other activities like downloading logs etc are bit slower.
Medal: Bitrise has again earned the Silver medal for Directory Caching
Secret Store
Bitrise has two tabs to manage the secrets and environmental variables. We can store out sensitive information there. Bitrise also has great documentation on how to configure secrets here.
Medal: Bitrise has earned the Gold medal for the secret store game.
Xcode & Apple Tools Upgrade
Bitrise is quick to upgrade the Apple developers tools as well. The Xcode 9 GM has been upgraded very quickly.
Medal: The Bitrise has earned the Gold medal for speed.
Open-Source Project Support
Bitrise is completely free for the open-source project, that’s the reason I got an opportunity to try the awesome features of Bitrise. It has hobby plan which is free and allows us to execute 200 builds per month. Thanks Bitrise for supporting open-source projects.
Medal: Bitrise has earned the Gold medal for the secret store game.
Learning Curve
Bitrise is easy to use and configure platform when used through the UI. There are concepts of workflows which make users life easier to configure the builds and integration with third-party tools. However, it’s tricky to understand whats happening under the hood while using the integration and workflows. The process of learning the command line interface for the Bitrise can be time-consuming but once learned it would be easy.
Medal: The Bitrise has earned the Silver medal for Learning Curve.
Documentation & Support
Bitrise has great documentation as well as discussion forums for the users. They provide good support on email as well as live chat. The live chat persons response might take up to an hour which isn’t ideal. Bitrise isn’t active on social media, they don’t respond to tweets or might be busy in something else.
Update: Bitrise corrected some of my points in the comments below and agree to improve social media support for customers and users. Upgrading medal from Silver to Gold
Medal: The Bitrise has earned the Gold medal for Documentation and Support.
Clean UI/UX
Bitrise has very clean and informative user interface. The user feels easy while setting up projects and configuring the workflows. The simple interface allows us to see logs and reports, however, reports are based on xcpretty format and purple colour offers can be sometimes very disturbing.
Medal: Bitrise has earned the Gold medal for the clean UI/UX game.
Price
The Bitrise has some good price plans but if the concurrent builds are grown in size then it would be considerable.
At the time of writing this post, Bitrise has this plan mentioned above.
Medal: Bitrise has allowed me to run builds for free which is great. I am giving the Gold medal for the price game.
Bitrise: Pros and Cons
Bitrise being the mobile-specific continuous integration service, it helps in building iOS apps, but there are some other things we need to consider while using Bitrise.
Pros
The benefits of using Bitrise are as follows
- Easy Setup
- Infrastructure as code using YML config file
- Support for workflows and custom scripts
- No need to write lot of custom scripts as there are workflow steps available
- Drag and drop of certificate and profiles
- Free for opensource projects with limited builds
Cons
There are some pitfalls using Bitrise which are as follows:
- Default build configurations might not work for some projects
- Test reporting mechanism isn’t great
- GUI based manual build configuration which is hard to repeat and recreate for other projects
- Bitrise is bit slow in downloading reports
- No inbuilt support for crash reporting or monitoring in the Dashboard
XCFit: Build Details
The build logs executed for the XCFit project are available here but you need to have an account to access those logs.
Bitrise Medal Summary
At this stage, Bitrise has earned 11 gold medals, 8 silver medals and 1 Bronze medal.
[table id=6 /]
Bitrise also has done well in the competition. Let’s keep an eye on how other players performing. We will publish the final medal table at the end of the competition. In the next post, we will see the performance of TravisCI. Continue Reading