Top 5 reasons for Continuous Integration failure

Shashikant Jagtap

Agile software development can’t be perfect without effective Continuous Integration process. Continuous Integration a.k.a CI is a process of continuously analyse, build, test and deploy software. Continuous Integration checks an internal quality of the code and tests the business logic of the product before we release it to the production. Ideally, we shouldn’t allow software to deploy to the product when the build is broken. You can read best CI practices here, however, Continuous Integration not always works for every agile team. Some agile teams take CI practices very seriously, some teams do it for sake of doing agile, some teams ignore it completely and some doesn’t have CI server setup yet.

There are various reasons, CI practices are being ignored within the team because business has different priorities, Product Owner doesn’t understand the importance of internal quality, testing processes and clean build. The Technical Manager can’t buy time to implement the CI practices or fix broken CI. The discussion between product and technology management fails to understand priorities of each other which ends up deploying the broken build to end users. The product people are happy because they delivered business value with broken CI. This approach can give a temporary satisfaction but it’s very dangerous. It can later throw severe bugs in the production which can heavily impact business. The severity of an impact is unpredictable ranging from the loss of money, loss of reputation or in extreme case, shutting down the business completely. However, even if Product and technology team agreed to invest a time and money to implement or fix CI issues,  some teams never succeed. Let’s discuss top 5 reasons of CI failures within the team and potential solutions to overcome the problems.

1. Wrong Selection of CI Server

There are various Continuous Integration tools available in the market The CI server solution can be self-hosted or cloud. You can get a whole list of CI servers here with recommendations. As of now, Jenkins was one of the popular CI server and people tends to use it blindly, so we have to adjust our project to work with Jenkins and the team has to compromise with the services that Jenkins offers. Now that situation has changed and there are various promising CI services arrived in the market. Looking at the wide range of CI servers in the market, it’s challenging to select one which suits our project need. The process of setting up an CI server takes time and investment. If you choose CI server without doing any research, and it doesn’t work well for  team then all the efforts you put in to get CI server will be in vein. The common mistake that management can do to save the cost is selecting a generic CI server or service for all platforms like Web, Desktop, or Mobile which doesn’t work quite well. Imagine, your application has website, iOS app and Android app so finding a generic solution might not work well. We have to be very careful about selecting CI server.

Potential Solutions

  • Research the market very well and evaluate few options on pilot projects. The Slant has already evaluated major CI servers with Pros and Cons of each, reading this would be good starting point for evolution process.
  • Look for the features like Pipeline support, Pipeline as Code, Docker Container Support, Platform Support, ease of use, availability etc while evaluating CI servers.
  • Don’t try to find a generic solution for all the platform to reduce cost. Each platform has different technical requirements and challenges.
  • Discuss with the team and ask for past experiences so that we don’t need to train engineers again.

2. CI Amateur Engineers

An engineer working in the agile team should have better coding skills but delivering a working software is more than just writing and testing code. It also involves setting up the appropriate environment to make sure we can deliver it easily. This requires strong command line and scripting skills for build automation as well as sound knowledge of build automation tools and dependency/package management tools. Recently, companies are moving the infrastructure to cloud so there is need to learn DevOps skills e.g Cloud Services like AWS, Azure, Heroku.  The provisioning tools like bash, Ansible,  Chef. The container services like Docker, Kubernetes. The most important is having knowledge of one the scripting language e.g Bash, Ruby or Python. It doesn’t mean that you should learn everything but you should learn everything for your platform. Let’s assume engineer is working on iOS application development, the engineer should know tools like Cocoapods,  Carthage, Swift Package Manager for the Dependency Management. Also build automation tools like Fastlane, Rake, Make and strong command over Apple’s native command line tools. An Engineer should aware of recent tools related to the relevant platform.

There are different types of engineers in the market,  some of them are good at writing native language code (Java, Objective-C, Swift)  as well have strong knowledge of DevOps/build automation tools. Some engineers just write code in an IDE ( Eclipse, IntelliJ, Xcode) and they never come out of an IDE and not capable of running commands from the Terminal.  Some engineers better at build automation tools but not that good at writing native language code. The CI amateurs are those who can not get out of IDE and not capable of using command line (scripting) tools. They prefer GUI tools to everything than command line or scripting. However, CI sever doesn’t have GUI interface to interact so everything has to be scripted.

If you have CI more amateur engineers in the team, CI practices never going to be successful. The CI amateur developers might end up writing poor quality build automation scripts which break all the time. The team end up with spending time on learning, improving build automation, switching between the CI servers rather than building features useful for the business.

Potential Solutions

  • Hire engineers who have basic knowledge of Continuous Integration and DevOps.
  • Train them if you already got CI amateur engineers. The good idea is to send them to external training or train them from in-house CI experts.
  • Hire expert on short-term contract to setup CI process and share knowledge about the processes.

3. Pipeline isn’t Coded

The most of the CI servers allow users to change the build configuration from the web interface. This approach is easy for engineers to create and edit the CI builds, however, changing build configuration frequently can cause a lot of issues like disabling important build step e.g disabling test runs or another important check. Also, everyone has permission to access the build machines which can cause confusion who did what and when. It takes longer to figure out build failures if engineer not aware of build setting changes. Frequent change in the CI server machine can cause problems and confusions within the team.

Potential Solution

  • Take control of build configuration by putting the config file,  bash script or similar file inside the source code
  • Avoid Manual Configurations on CI server
  • Restrict Access to CI server machine to few people and give them the responsibility to manage the server.
  • Do not allow user to modify particular build steps

4. Poor Quality Build Server Machines

The developers working in the team frequently check-in code in the source control system which triggers build on CI sever. It means CI server machine is continuously running some jobs which can involve heavy tasks like downloading dependencies from remote servers, backing up databases, running Docker containers etc. In order to perform those task effectively, CI server must be fast and reliable and connected to the network. Using poor quality server waste everyone’s time because build takes too long to finish, intermittent test results and frustrated engineers.

Potential Solutions

  • Get fast and reliable server for Continuous Integration, avoid cheap servers
  • Never leave CI server machine on WiFi.
  • Never install unnecessary software on CI server machine
  • Smartly share CI server machine by assigning to specific job
  • Never install any software manually
  • Avoid giving GUI access to machines. SSH Access should be sufficient

5. Lack of Management Support

The project management plays a key role in enforcing CI practices within the team. The management should set strict guideline about the CI builds and should have zero tolerance for the broken build. The software shouldn’t be released with the broken build in any case. The broken CI build be treated as emergency and whole team should work on it to fix it before we commit any further code. The technical manager or relevant person from management can set this guideline and implement those guidelines strictly within the team. The managers without any understanding of Continuous Integration are likely to carry on with feature development regardless of code quality. The Continuous Integration practices won’t be successful under such management.

Potential Solutions

  • Setup CI practices with team and apply it strictly within team
  • Train project managers with CI practices if they already not

 Conclusion

It’s challenging to implement Continuous Integration practices within an agile team but following some strict rules and avoiding common mistakes can make CI processes much effective. What are your experiences with CI Server? Do you feel your CI practices working well? Weigh in with a comment below!