Automation DevOps Featured iOSDev

iOS CI Olympics-Player Two: Bitrise

Before diving into this article, make sure you take a look at the intro article on iOS CI Olympics game here.

In the introductory post on iOS CI Olympics, we have selected the few player and few games for this competition. We will be evaluating the player one by one on the basis of their performance in the each game. The first player was BuddyBuild and we have seen its performance in the previous post. In this post, the player is Bitrise, another mostly heard iOS CI service.

Bitrise

In the iOS Continuous Service Olympics competition, the second candidate selected is Bitrise. There are many articles on medium and lot of good words on Twitter about the Bitrise. Like BuddyBuild, the Bitrise is also designed for the mobile continuous integration which supports iOS and Android platforms. Bitrise also got some good clients like ABB, Foursquare, FOX, TNT etc. The little of the background of Bitrise organisation

Company: Bitrise
Founded: 2014
Headquarters: Budapest
Description: Mobile continuous integration and delivery platform, with dozens of integration.
Founders:Categories: AndroidDeveloper ToolsiOSAppsDeveloper Platform
Website:https://www.bitrise.io/

In this post, I will share my first impressions of getting started with Bitrise for my favourite pet project XCFit using the iOS CI Olympics games.  So Let’s get started with our games

SignUp & Project Setup

Bitrise was good in connecting to the Github repositories. Just need to select the repository that we want to build on Bitrise. I have selected XCFit repository to build on Bitrise.

Now that, we have selected the app that we want to build, the next step that Bitrise automatically do is scan the repository with specific GitHub branch. In case of XCFit, its master branch

Medal: Bitrise has earned the Gold medal for the project setup game

Default Build Configuration

Once, we select the repository and branch to build then Bitrise checkout the source code from the repository and find out the iOS apps and stats building the project using the default build configuration.

As mentioned earlier,  XCFit has an example app inside XCFit-Example directory,  that has Cartfile  to build the project using Carthage. The default configuration of Bitrise was failing to find the   Cartfile located inside the XCFit-Example directory. The build was failing using the default configuration.

It took a lot of time for me to figure out the Bitrise workflows and how Carthage is being used. In the end, I have deleted the Carthage workflow step and added the custom script to fix this issue. Apart from that, the Bitrise process was great.

Medal: Bitrise has earned Silver medal for the default build configuration game.

Build Execution Process

Bitrise starts to build quickly, in my case within few seconds after commit or as soon as possible. The build execution logs are shown very clearly and cleanly. The logs are shown in the xcpretty format which is readable and there was a button to download full logs. Bitrise also has a separate tab for the app artefacts and test reports. However, this doesn’t impress me, as Bitrise is just reusing xcpretty for displaying logs and creating reports. What will happen if xcpretty stops working in the future?  I am not sure they have alternative plan to overcome this potential issue

Update: After having conversations with Bitrise in the comments below, It seems like Bitrise can have multiple reporting formats and we can switch between them. Upgrading from Bronze to Gold medal.

Medal: Bitrise has again earned the Gold medal for the build execution game

 

Build Artefacts and Test  Reporting

After the build has finished, the entire logs are available for the download. There are two tabs in the build execution, one shows the entire logs and other shows the apps and artefacts. The test reports are generated using xcpretty HTML reporter format which isn’t that helpful. There are no the code coverage reports for the unit tests.

Medal: Bitrise has again earned the Silver medal for Build Artefacts and Test reporting game

Real Device Testing

Bitrise has support to execute iOS test using simulators but there isn’t any way to run the tests on real devices both for Android and iOS apps. Bitrise uses vagrant virtual machines so it would be difficult to add support for running UI tests on real devices.

Update: After having a conversation with Bitrise representative in the comment, it’s possible to configure tests on real devices for Android and trigger tests on third-party services like Saucelabs, AWS Device Farm. Upgrading from Bronze to Silver medal

Medal: Bitrise has again earned the Silver medal for the Real Device Testing game

 

 

Build Customisation

The default build configuration of Bitrise can be easily overridden using the workflows features of Bitrise. There are various workflows available to customise the build steps. There is also the possibility to add your own custom script just in case, we need something to do in between the workflow. The workflows can be re-arranged accordingly. The environmental variables are pretty common in the Continuous Integration system, Bitrise also has the ability to define the environmental variables with its values.

The great feature of Bitrise is that it allows us to code the entire workflow using  bitrise.yml file. This file can be used on the local machine of the developer as well.

 

Medal: Bitrise has earned the Gold medal for Build Customisation game.

Parallelisation & Concurrent Builds 

Bitrise has price plans per concurrencies so there might be the possibility to execute the concurrent build per plan. I am not sure what would be the maximum limit for the concurrent build. Bitrise dashboard doesn’t show any option to select the multiple simulators so I don’t think there is the possibility to parallelize the test execution as well.

Update: As per the comment in the post, Bitrise can go up to the maximum number of concurrencies as users pay more. Please read the comment below which explains the details which I couldn’t understand in the first place. I have upgraded Bitrise to Gold medal

 

 

 

Medal: Bitrise has again earned the Gold medal for the Parallelisation game

Build Pipelining

Bitrise has the concept of workflows which allows us to define the steps or phases within a particular build, however, the pipeline should allow us to execute or configure the builds in parallel or sequential manner. Using Bitrise workflows smartly we can achieve the pipelining of the iOS deployments but its limited and workflows cannot be run in parallel.

Medal: Bitrise has again earned the Silver medal for Build Pipelining game

 

 

Infrastructure As a Code

Bitrise is great in allowing the user to configure the workflows using UI as well as providing the command line interface to define the build steps. This allows the user to test the Bitrise workflows in the local machines before they push to the source control. Bitrise has also awesome documentation to get started with it.

Medal: Bitrise has earned the Gold medal for Infrastructure as Code game.

 

 

Directory Caching: CocoaPods & Carthage

CocoaPods and  Carthage are the two most used dependency manager for iOS projects. Both dependency managers check out the dependency source code and build the frameworks in Pods and Carthage directory. This is a very time-consuming process to do for every build.  We need to cache those directories to speed up the build execution. Bitrise supports the caching of both Carthage and Cocoapods but it requires configuration of workflows.  The forum has the solution to cache the Cocoapods and Carthage directories to speed up the builds.

But there is no official parameter to cache the directories.

Update: As per the comment from Viktor below, Bitrise has the clean mechanism of caching the both Carthage and CocoaPods. Bitrise has added cache: pull and cache: push mechanism that applied to the default build. Read this discussion to know more here. Bitrise got Gold medal for this game

Medal: Bitrise has again earned the Silver medal for Directory Caching 

 

Code Signing & Distribution

The code signing with Bitrise seems to matter of giving an access to Apple Developer Portal or upload the certificates to Bitrise. I think everything else like downloading provisioning profiles and code signing apps has been taken care by Bitrise However, code signing features haven’t been tested as part of the XCFit project.

Medal: The areas that I haven’t fully tested with XCFit projects, I will be giving the Silver medals by default to untested but available features

Third Party Integrations

Bitrise seems to have so many third-party integrations available in the workflow. There are various steps which cover the third party apps like SwiftLint, SonarQube, AWS etc.

 

Medal: Bitrise has earned the Gold medal for the third party integrations game.

Speed

Bitrise build usually starts and runs quickly. The executing the build was fast enough but other activities like downloading logs etc are bit slower.

Medal: Bitrise has again earned the Silver medal for Directory Caching 

 

Secret Store

Bitrise has two tabs to manage the secrets and environmental variables. We can store out sensitive information there. Bitrise also has great documentation on how to configure secrets here.

Medal: Bitrise has earned the Gold medal for the secret store game.

 

Xcode & Apple Tools Upgrade

Bitrise is quick to upgrade the Apple developers tools as well. The Xcode 9 GM has been upgraded very quickly.

Medal: The Bitrise has earned the Gold medal for speed.

Open-Source Project Support

Bitrise is completely free for the open-source project, that’s the reason I got an opportunity to try the awesome features of Bitrise. It has hobby plan which is free and allows us to execute 200 builds per month. Thanks Bitrise for supporting open-source projects.

Medal: Bitrise has earned the Gold medal for the secret store game.

Learning Curve

Bitrise is easy to use and configure platform when used through the UI. There are concepts of workflows which make users life easier to configure the builds and integration with third-party tools. However, it’s  tricky to understand whats happening under the hood while using the integration and workflows. The process of learning the command line interface for the Bitrise can be time-consuming but once learned it would be easy.

Medal: The Bitrise has earned the Silver medal for Learning Curve. 

Documentation & Support

Bitrise has great documentation as well as discussion forums for the users. They provide good support on email as well as live chat. The live chat persons response might take up to an hour which isn’t ideal. Bitrise isn’t active on social media, they don’t respond to tweets or might be busy in something else.

Update: Bitrise corrected some of my points in the comments below and agree to improve social media support for customers and users. Upgrading medal from Silver to Gold

Medal: The Bitrise has earned the Gold medal for Documentation and Support. 

 

Clean UI/UX

Bitrise has very clean and informative user interface. The user feels easy while setting up projects and configuring the workflows. The simple interface allows us to see logs and reports, however, reports are based on xcpretty format and purple colour offers can be sometimes very disturbing.

Medal: Bitrise has earned the Gold medal for the clean UI/UX game.

 

Price

The Bitrise has some good price plans but if the concurrent builds are grown in size then it would be considerable.

At the time of writing this post, Bitrise has this plan mentioned above.

Medal: Bitrise has allowed me to run builds for free which is great. I am giving the Gold medal for the price game.

Bitrise: Pros and Cons

Bitrise being the mobile-specific continuous integration service, it helps in building iOS apps, but there are some other things we need to consider while using Bitrise.

Pros

The benefits of using Bitrise are as follows

  • Easy Setup
  • Infrastructure as code using YML config file
  • Support for workflows and custom scripts
  • No need to write lot of custom scripts as there are workflow steps available
  • Drag and drop of certificate and profiles
  • Free for opensource projects with limited builds

Cons

There are some pitfalls using Bitrise which are as follows:

  • Default build configurations might not work for some projects
  • Test reporting mechanism isn’t great
  • GUI based manual build configuration which is hard to repeat and recreate for other projects
  • Bitrise is bit slow in downloading reports
  • No inbuilt support for crash reporting or monitoring in the Dashboard

XCFit: Build Details

The build logs executed for the XCFit project are available here but you need to have an account to access those logs.

Bitrise Medal Summary

At this stage, Bitrise has earned 11 gold medals, 8 silver medals and 1 Bronze medal.

GAMEGOLD
Total 13G 7S 0B
Sign Up & Project Setup
🥇
Default Build Configuration
🥈
Build Execution 
🥇
Build Artefacts & Test reporting 🥈
Real Device Testing 🥈
Build Customisation 🥇
Parallelisation & Concurrent Builds 🥇
Build Pipelining 🥈
Infrastructure As a Code  🥇
Directory Caching: CocoaPods & Carthage  🥇
Code Signing & Distribution 🥈
Third Party Integrations 🥇
Speed 🥈
Secret Store
🥇
Xcode & Apple Developer Tools Upgrades 🥇
Open-Source Project Support 🥇
Learning Curve
🥈
Documentation & Support
🥇
Clean UI/UX 🥇
Price 🥇
Magic Medals TBC

Bitrise also has done well in the competition. Let’s keep an eye on how other players performing. We will publish the final medal table at the end of the competition. In the next post, we will see the performance of TravisCI. Continue Reading

  • > However, this doesn’t impressed me, as Bitrise is just reusing xcpretty for displaying logs and creating reports. What will happen if xcpretty stops working in the future? I am not sure they have alternative plan to overcome this potential issue

    There already is an option for this 😉 You can set the output format in the Xcode steps. Default is xcpretty and a second option is to simply present the xcodebuild generated output. So even today you can simply switch to that. The steps are also open source so you’re free to add any alternative formatter, as well as xcpretty is also open source and so can be forked.

    • Hey Viktor, Thanks for your comments mate! Really appreciate, this evaluation is on the basic of my first impressions of CI so treat it as User Testing 🙂 I might have misjudged something or haven’t discovered the features. Glad that you have read the post and provided your feedback 🙂 I will update the post as well as medal table accordingly 🙂 Feel free to reach out to me if any question about the post. Thanks for such a awesome product !

      • > this evaluation is on the basic of my first impressions of CI so treat it as User Testing :

        Sure, definitely makes sense and thanks for the post! Just wanted to highlight these things which might not be that simple to find – we’ll fix that, and your post helped a lot to see where docs/landing page is lacking! 😉

  • > Real Device Testing

    You’re free to use any real device testing service. See e.g. https://discuss.bitrise.io/t/how-to-run-android-tests-on-firebase-testlab/2192

    There are also build steps, e.g. for Amazon Device Farm, as well as for Open STF if you don’t want to use a script.

    • Yes, But this seems to for Android devices but you are right we can trigger tests on real devices of other platforms like AWS Device Farm, SouceLabs or Testdroid etc from Bitrise 🙂

  • > they don’t respond to tweets or might be busy in something else.

    we always do 🙂

    • No they don’t ! At least for me. I will keep tweeting and see if they responds 🙂 Anyway never mind, everyone has priorities ..

      • Hmm.. do you have a tweet link? I’d check that – might have slipped through 🙂

  • > Directory Caching: CocoaPods & Carthage
    > Bitrise supports the caching of both Carthage and Cocoapods but it requires configuration of workflows.

    This is no longer required. If you use our CocoaPods and Carthage steps as well as the latest Cache steps these should “just work”. New projects are also auto configured with these steps and require no manual config.

    See e.g. https://discuss.bitrise.io/t/how-to-cache-cocoapods-dependencies/193 -> “If you use v1.0 or newer of the Cache steps and the latest CocoaPods Install step then that’s all you have to do.” (and this should be in the default configuration generated when adding an iOS project now).

    • Hi @viktorbenei:disqus Thanks a lot for pointing this up. I wasn’t aware of this feature. Would be great to have cached directories but do you check if something is changed and update the required dependency accordingly ?

      • If you use our steps those can handle updates too. If you use other tools instead of our steps, eg fastlane, that works too but you’ll have to configure the cache for the tool.

  • > Bitrise has price plans per concurrencies so there might be the possibility to execute the concurrent build per plan. I am not sure what would be the maximum limit for the concurrent build.

    I see your point, this should definitely be easier to see on the pricing page. A short description: once you register and you go to the pricing page or go to your account settings / billing page you can set how many concurrencies you need with a simple slider. Each concurrency costs the same, so 4 concurrencies is 2x the price of 2 concurrencies for example.

    It’s on the pricing page too, but the design should be better, might not be obvious if you did not scroll a bit down on the page: https://uploads.disquscdn.com/images/0600c18967087095b03499628ad426a5456108878200cfce0691cef587b79aa1.png

    Concurrent builds are automatic if you have more than one concurrency in your plan; concurrency means how many of your builds can run at the same time.

    I hope this explains it, if you have any questions just let us know! 😉

    • That makes much clearer @viktorbenei:disqusThanks for explanation. As per other CI services e.g Travis can run 10 concurrent build CircleCI can go unto 12x concurrency. Does Bitrise has that limit ? I think, there should be. What will happen if there are millions of concurrent build triggered 🙂 Sorry Joke apart but this information you provided very useful .. Thanks again 🙂

      • Sure! Default limit is 5 for personal and 10 for organization subscription. If you try to set the slider higher the button changes to “contact us”. We can usually bump the limit real quick, it’s just to be sure we have the capacity.

        • Thanks @viktorbenei:disqusI have updated the post accordingly with your information, Bitrise went 2 gold medal up 🙂 Thanks for clarification !