Before diving into this article, make sure you take a look at the intro article on iOS CI Olympics game here.
In the introductory post on iOS CI Olympics, we have selected the few player and few games for this competition. We will be evaluating the player one by one on the basis of their performance in the each game. In this article, the player is TravisCI, a pioneer in providing free continuous integration to open-source projects.
TravisCI
In the iOS Continuous Integration Server Olympics competition, the third candidate selected is TravisCI, the most used cloud-based continuous integration for open-source and private projects. It’s truly the home of the open-source testing. As mentioned on the TravisCI page there are about 900K opensource projects and 600K users running the continuous integration on TravisCI. It’s deeply integrated with GitHub and setting up an open-source project on TravisCI is a minute away from developers. The little bit information of TravisCI is as follows
Company: TravisCI
Founded: 2011
Headquarters: Berlin
Description: Hosted Continuous Integration Platform
Founders: Konstantin Haase, Mathias Meyer, Fritz Thielemann, Sven Fuchs, Josh Kalderimis
Categories: Web Apps, Enterprise Software, Information Technology
Website:http://travis-ci.com
In this post, I will share overall experiences with TravisCI for my favourite pet project XCFit using the iOS CI Olympics games. So Let’s get started with our games
SignUp & Project Setup
The Travis is made up for Github, if you got Github account then registering up with Travis is so easy. Just use your Github credentials and you are inside the Travis.
Like other CI servers, Travis won’t setup project by himself. There are the couple of steps we have to perform to get going. We need to sync the GitHub project and enable the repo to build on TravisCI. We also need to add .travis.yml file to the root of our project. Travis will guide you to complete the project setup.
The content of the configuration file would be very simple as mentioned in the getting started guide here. In case of XCFit project, I added something like this to configuration file to get going
|
1 2 3 4 |
language: objective-c osx_image: xcode9 script: - echo "Hello Travis" |
Once, code is pushed to the Github repo then my project started building on Travis.
Medal: Travis has earned the Gold medal for the project setup game
Default Build Configuration
There is no concept of default configuration in Travis. The user needs to know what they want to build on Travis and configure the .travis.yml file accordingly. Travis will follow your instructions and build accordingly. You will have full control on the project. No guessing, no assuming, whatever you script in the config file that will be executed on Travis. In case of XCFit project, I want to navigate to XCFit-Example directory, run Carthage to download dependencies and execute my test with fastlane. Before that, I probably need to install Fastlane gem and cache Carthage directory for faster build execution. The final script will like this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
language: objective-c osx_image: xcode9 cache: directories: - XCFit-Example/Carthage before_install: - gem install fastlane --no-ri --no-rdoc --no-document install: true script: - cd XCFit-Example - carthage bootstrap --platform iOS - fastlane scan -s XCFit-Example |
Now that, I have scripted what I would like to run. The Travis will take care of executing the builds from now.
Medal: Travis has earned the Gold medal for the default build configuration game.
Build Execution Process
As soon as, we push the commit to Github repo, Travis will trigger the build. We can configure the builds for branches or pull requests. The Travis will boot brand new VM, check out the source code and execute the configure scripts. The booting of the VM may take some time to start the actual build, so the build will be bit slower but as soon as build started the execution process is very smooth. Travis log shows the details of the every step and collapses it when the step is executed, this reduces the log noise.
The above logs show that how Travis has executed all the steps that we configured for the XCFit project.
Medal: Travis has again earned the Gold medal for the build execution game
Build Artefacts and Test Reporting
Travis doesn’t keep the build artefacts for the build, we need to upload the artefacts to some other services like AWS S3 bucket or Artefactory or similar. Travis will kill the VM once the build has been finished.
The Travis has nothing other than xcpretty–travis-formatter to display the test results in the build logs. Unfortunately, we can’t get better test reporting with Travis.
Medal: Travis has earned the Bronze medal for Build Artefacts and Test reporting game
Real Device Testing
Travis runs the tests inside the macOS virtual machines that get killed once the build is finished. There is no possibility to add real device testing support with Travis. However, we can write custom scripts to trigger the test on real devices provided by third-party providers like AWS Device farm or Saucelabs
Medal: Travis has earned Bronze medal for Real Device Testing game.
Build Customisation
The build settings in the Travis are very limited which includes turning of branch buildings, cancelling redundant builds and setting up environmental variables.
The environmental variables are pretty common in the Continuous Integration system, Travis has an ability to define the environmental variables with its values that can be used while building. The user can define own environmental variables on top of Travis default environmental variables.
Medal: Travis has earned Silver medal for Build Customisation game.
Parallelisation & Concurrent Builds
Travis has a cool feature that allows us to define the matrix to be run in the parallel. This feature is called Build Matrix. We can define the steps to run in parallel. e.g If we want to run the test in different simulators then we can define the matrix with the name of simulators or iOS versions as environment variables. The builds will run in the parallel without taking extra time. The Travis can also give us concurrency but it involves cost. Travis can go upto 10 concurrent builds as per their pricing plan.
Medal: Travis has earned the Gold medal for Parallelisation and Concurrent Builds game.
Build Pipelining
Travis has recently introduced the new feature called Build Stages that allows us to group the jobs and run them in the phases. One stage can run many jobs in parallel but another stage can only be triggered once the previous stage passes. e.g we can define the stage to run the unit and UI tests like this
|
1 2 3 4 5 6 7 8 9 |
jobs: include: - stage: XCTest script: - fastlane scan -s UnitTests - stage: XCUITest script: - fastlane scan -s UItests |
I wrote the detailed article on Travis build stages here explaining its pros and cons. This feature looks awesome.
Medal: Travis has earned the Silver medal for Build Pipelining game.
Infrastructure As a Code
TravisCI is designed to work as an Infrastructure as Code. The whole build pipeline has to be coded in the .travis.yml file. There is very minimum UI configuration for the build allowing us to reduce human errors. There are code changes required to change the build configuration.
Medal: Travis has earned the Gold medal for Infrastructure as Code game.
Directory Caching: CocoaPods & Carthage
CocoaPods and Carthage are the two most used dependency manager for iOS projects. Both dependency managers check out the dependency source code and build the frameworks in Pods and Carthage directory. This is a very time-consuming process to do for every build. We need to cache those directories to speed up the build execution. At the time of writing this post, Travis support for caching the CocoaPods or Carthage directories. We have used the settings in the config file to cache the Carthage directory to speed up the builds. I wrote a detailed article on how to cache Carthage using Travis here. Travis has huge support in caching the CocoaPods and Carthage
Medal: Travis has earned Gold medal forDirectory Caching game.
Code Signing & Distribution
Travis doesn’t have any inbuilt support for the code signing of the iOS apps. The user has to script all the required steps from adding distribution certificate to keychain till uploading a binary to iTunes connects. This requires strong scripting skills. This blog post explains most of the areas to automate iOS deployments on Travis.
Medal: The areas that I haven’t fully tested with XCFit projects, I will be giving the Silver medals by default untested but available features.
Third Party Integrations
Travis has no official third-party support for any service. However, we can use custom scripts to integrate Travis with other services. e.g deploying a build to Fabric. There might some cases where Xcode image of Travis has some pre-installed tools.
Medal: Travis has earned the Bronze medal for Third Party Integrations game.
Speed
TravisCI boots fresh VM for the new build. The process of booting, checking out the source code and installing pre-defined software can take a long time. The actual build will start once all the setup has been some. Travis builds are bit slow. It might be great for the paid version but open-source version takes a bit of time to finish the build.
Medal: Travis has earned the Silver medal for speed.
Secret Store
Travis has an option to store all the secret in the form of an environmental variable that can be accessed from the settings page of the Travis build.
We can inject the access tokens, encryption keys and all sort of sensitive information here. Travis will store then and won’t print as part of build execution.
Medal: Travis has earned the Gold medal for Secret Store game.
Xcode & Apple Developer Tool Upgrades
Travis is usually fast in upgrading the Xcode images and other developer tools. Travis has blogs to let the users know that new images are available to use. In case of Xcode9 Tarvis has that images since beta 1.
Medal: Travis has earned the Gold medal for fast Upgrades of Apple developer tools.
Open-Source Project Support
Travis is one of the pioneers CI service who is serving the open-source community since long. As mentioned earlier, there are 900K open-source projects and 600K users are using Travis for their open-source projects. The numbers tell the popularity of the Travis.
Medal: Travis has earned the Gold medal for Open-Source Project Support.
Learning Curve
Travis doesn’t configure the build by himself. The user has to script all the build requirements in the config file. There is a little bit of learning curve when we get going but once learned its fun. The user needs to learn the scripting Travis features and integration to the third party but it gives all the power to the user.
Medal: Travis has earned the Silver medal for Learning Curve.
Documentation & Support
Travis has amazing user documentation which guides the user from getting started to configuring complex builds. There is all sort of information with example code. Travis also showed how to use third-party tools with Travis in the documentation. Travis runs a blog to keep users updated with the latest changes in the infrastructure. Travis has the status page to update users if there are any known issues.
Medal: Travis has earned the Gold medal for Documentation and Support.
Clean UI/UX
Travis also has clean and intuitive UI that helps users to set up the projects easily and without any pain. The features are easy to discover with the clean UI.
Medal: Travis has earned the Gold medal for Clean UI/UX.
Price
As Travis is offering free support for the number of open-source projects. It also has four paid plans that suit from small scale to big enterprises. At the time of writing this post, Travis has following plans
The Travis has enterprise plan which gives us 10 concurrent builds.
Medal: Travis te deserves the Gold medal for the price game.
TravisCI: Pros and Cons
As of now, we have seen the features of the TravisCI in brief, now the time is to decide what’s good and not good with TravisCI. There are certainly some great features that TravisCI offers but there are some limitations as well.
Pros
The benefits of using TravisCI are as follows
- Travis is free for the open-source projects
- Deep integration with Github allows easy setup of repository on CI
- Ability to use config files gives more power to users to control the build execution
- Travis has an excellent web interface.
- The new features of Travis like build Matric and build stages are very promising.
- Travis has huge community
Cons
There are some pitfalls using Travis which are as follows:
- Travis only supports Github projects, no supports for other SCM
- Sometimes Travis VM takes edges to boot which slows down the build
- Steep learning curve for the new users.
- No support or assistance on Code signing iOS apps
- no native support for the third-party tools
XCFit: Build Details
The Travis build logs executed for the XCFit project are available here.
TravisCI Medal Summary
[table id=7 /]
Travis medal counts looks impressive. In the rest of the Olympics competition, we will see how other players will perform. We will publish the final medal table at the end of the competition. In the next post, we will see performance of the Nevercode. Continue Reading ...