Xcode Server: Top 10 Limitations of XCS for iOS Continuous Delivery

Just after Apple announced Xcode Sever a.k.a XCS inbuilt with Xcode 9, I wrote a detailed post on how to setup Xcode Server for iOS continuous integration with Xcode 9. The post also has example project to demonstrate new features of Xcode Server. The post mentioned some of the key features including inbuilt Server, Headless and parallel testing, Automatic Code signing and device provisioning. The post is originally published on my personal blog and reposted on Medium and DZone as well. However, sometime after, I started receiving comments about the limitation of Xcode Server to achieve continuous delivery of iOS apps. I also experienced some issues while using Xcode Server for CI. Some of them are known and some of them were unknown. I thought it’s worth sharing in the post, we will discuss the top limitations of Xcode Server.

Current Limitation of Xcode Server

There is the list of various issues related to Xcode Server here but we will cover top 10 issues with Xcode 9 and Xcode Server. Note that, all these issues mentioned below are at the time of writing this post.

1. Pull Request Testing

Currently, there is no way to test Github pull requests. As many CI servers like TravisCI can build the iOS project as soon as Pull requests are created so that developers can get early feedback. However, with Xcode Server, this isn’t possible at the moment. I was under impression that Xcode Server isn’t built just for Github but looking at the fact that Xcode 9 has very tight integration with Github. Apple should consider this problem as well. This is badly missed feature in Xcode Server. This product won’t be complete without this feature.

Previously there were some tools provide a workaround. The tools like Buildasaur or XBot Builder can be used to watch GitHub repositories for pull requests. As soon as the pull request created, these tools used to create Xcode Bots and update GitHub with results. However, things have been changed a lot. Those tools no longer work with the latest version of Xcode Server.

Another workaround would be to create a GitHub webhook to watch pull request, create Xcode Bots using Xcode Server API and update GitHub pull request with the result of integration. This involves heavy scripting. The most frustrating part is users can’t create Xcode Bots with Xcode Server APIs at the moment which we will discuss later. So currently there is no workaround to test GitHub pull requests

2. Bot Creation with Xcode Server API

Currently, Xcode Server is throwing internal server error while creating bots using POST request to Xcode Server API. I tried myself as well as some other users also facing the same issue. If that’s the genuine issue then, I would say this is the most critical issue. The use of Xcode server API became pointless when we cannot create bots using POST request. This also means we can’t test pull requests using custom scripts mentioned above.

3. Scalability

One of the major problems of using Xcode Server at the enterprise level is its scalability and distribution. Xcode Server seems to be the good option for the small team with dedicated singe macOS server or few servers. While managing multiple macOS Servers with Xcode Server would be bit tricky. As Xcode Server has few limitations

  • Xcode Bots cannot be assigned to server dynamically.
  • Managing multiple macOS Servers needs configuration management, the manual setup would be the nightmare.
  • With Xcode Server, we can not utilise idle servers and share loads from busy servers

If Xcode Server can give us the flexibility of dynamically assigning servers for Bots then it would be the very scalable approach.

4. Lack of support to upload IPA to iTunes Connect

At the moment, when archive option is selected for Xcode Bot, Xcode Server creates an archive and IPA file that can be installed on the device or uploaded to iTunes Connect. However, the process stops there. There is no inbuilt option to upload an IPA to iTunes Connect for TestFlight testing. There seems to be the option that allows us to submit an archive to App Store but we miss TestFlight step there.

As a workaround, we have to download IPA file and manually upload to iTunes Connect using Application Loader or local Xcode or using  agvtool command line utility. After that, the build will be available for testing on TestFlight.

5. Xcode Bot BackUp

Xcode Server doesn’t have any clean mechanism to maintain Xcode Bots in case of resetting CI server. Personally, I would love to tear down and rebuild Mac minis at regular intervals so that we avoid the mess getting accumulated on the server machine. Also in case of Xcode upgrade, we need to back up the bots for the newer version. Xcode Server also has an option to reset all the data, but backing up the bots aren’t straightforward.

Ideally, we can write script or use configuration management tools like Ansible or Chef to recreate entire environment including bot creation with recent configuration. However, this can’t be achieved at the moment as we are blocked by Bot creation issue. Some people mentioned that we can copy entire /Library/Developer/XcodeServer directory and paste with the new version. I’m not entirely sure how clean that approach that would be as there might be some changes in database or users or something else. The related thread here.

6. Keeping last few integration

Xcode Bots which build, test and archive an iOS app eats a lot of disk space. In my experience, every integration takes at least 500MB. It means there are chances that we ran out of disk space very soon. The related thread is here.

Ideally, there should be the mechanism to keep last X number of integration and delete all others. Like other CI server which does same things. They keep last few builds and pinned or favourite builds and cleans up others. It would be great if Xcode Server has similar kind of feature.

As a workaround, we have to manually delete the integrations using Xcode Server API or delete the integration assets on the Xcode Server.

7. Lack of third-party integrations

Xcode Server has the feature of adding pre-integration and post-integration scripts. By using this feature, we can integrate with any possible third-party services like Slack, HipChat, Github, Fabric etc. This approach is very time consuming and most of the iOS developers don’t have scripting skills to write those integrations. It involves analysing the third-party API and writing scripts in another language like bash, ruby or python. I am not sure about any other third party but Apple should definitely consider integration with Github.

8. Tagging builds in Github

One of the comments on my blog post stating that Xcode Server lacks the ability to tag the builds on Github.  As Xcode Server doesn’t have any third-party integrations, it’s not possible to tag the builds on GitHub but it can be easily achieved by the little bit of scripting.

9. Sharing Provisioning Profiles

When we have multiple Xcode Servers machines, sometimes its hard to keep track of the provisioning profiles and Xcode Server fails to create an archive. Some of the users already experiencing the problems and not entirely sure whats missing even they allowed Xcode Server to manage all the certificates and provisioning profiles. Xcode Server can automatically handle the code signing by clicking  Allow Xcode Server to manage my certificates and profiles but sometime you may get Archive Failed error. It may be because Xcode Server doesn’t have enough details about provisioning profiles needed to code sign iOS app. There is hack mentioned here where we need to turn off automatic code signing and download all the provisioning profiles using Fastlane sigh in the pre-integration script.

10. Parallel Testing issue with multiple simulators

There were various issues reported on the parallel testing feature of Xcode Server.  While selecting multiple simulators for testing on Xcode Server, the build fails but test passes locally. This tweet shows the error which says unknown state for the simulator. I have also faced similar error and I solved that sticking to the only one simulator.

Conclusion

Although Xcode Server is now the great solution for getting Continuous Integration up and running for the iOS apps, still there are some limitations that are painful to Xcode Server users which need to be considered as well. There are workarounds for those limitations but it would be great if that’s been fixed by Apple. What are your experiences, have you faced any pain from Xcode Server that I have missed in this post? Share in the comments below.