Machine Learning and Artificial Intelligence are the booming topics in the software industry at the moment. The use of AI/ML for testing apps also becoming the trend as there are many companies working to solve the testing challenges using AI/ML. The companies like mabl, test.ai, Bitbar, Applitools + many more are already started to roll out the products which use AI/ML to support testing of software applications. Just organised London Test Automation in DevOps meetup on the topic of Machine Learning in Testing sponsored by TestCraft where I have given a talk on Can CreateML used for Testing iOS Apps. In this post, I will explore the same topic in details to see if Apple’s brand new CreateML framework can be used to help in testing iOS apps.
Goal
The goal of this post is to teach how to train the machine learning model with CreateML with step by step process and automate the model training process. We will also explore how this trained model can be used for testing iOS apps. We will use the Apple News app for this demo. Apple News app has a home screen and spotlight screen we will take some of the pictures of both screen for training and we will also take some images for testing. I have created a sample data with a few images here.
What is CreateML
Apple announced CoreML back in 2017 but there was no way to train the machine learning models unless the developer uses third-party frameworks like TensorFlow or IBM Watson. In 2018, Apple announced Create ML framework to train the machine learning models that can be used with CoreML in any iOS app. There is no need to learn TensorFlow, IBM Watson or another third party framework.
Create ML is a machine learning framework in Swift which can be used to train the machine learning models using native Apple technologies like Swift, Xcode, and Other Apple frameworks. This means machine learning is possible within the Apple ecosystem without any third-party service. CreateML technology has the following features
- A simple approach to training the model using images, texts, and tabular data
- Models still can be trained using a complex algorithm if you are expert
- Machine learning models can be created using Xcode Playground, Swift, and native Apple framework
- CreateML models can be easily integrated into apps, just by drag and drop
- CreateML is powered by Mac and its available to use from macOS Mojave
- The process of creating and training module can be automated using Swift scripts.
Training ML Models Using Xcode Playground (Manual)
The manual process of training the machine learning models is very simple. You just need to have training and testing data. Training data to train the model and testing data is to test the model’s accuracy. The testing data should be unknown or less known to the model. We can have data in the form of images, CSV or JSON files. In this demo, we will train the model with images of Apple News iOS app. Get some images of the home screen and spotlight screen of the Apple News app, I have created a sample data in the Training and testing directory here. Once, we have data ready, open Xcode 10 on macOS Mojave and follow the following steps.
- File –> New –> Playground
- Select the macOS tab and Blank template
- Give a name to the Playground and Save it
Once in the playground, we can write the following code.
|
1 2 3 4 |
import CreateMLUI let builder = MLImageClassifierBuilder() builder.showInLiveView() |
Once, this code is written in the Xcode playground then turn on the assistance editor from View –> Assistant Editor –> Show Assistance Editor and Run the Playground using CMD+R or using the play button from the left-hand side. You will see the ImageClassifier assign to drag the Training data as shown below.
Now that, we can drag the training data directory in the area shown in the Xcode playground. This is a process all the training images and shows the results once the image processing is done. In the end, you will see the model accuracy as shown below.
Now that, we have successfully trained the model and it’s time to test model with some images that have never been exposed to the model. We have some images kept in the testing directory to test the model. Now CreateML is asking for the testing data, we will drag the testing directory over there to check the accuracy of the trained model. Once we drag the testing data, the model will give us information about the accuracy, confusion metrics and precision-recall. We can see that information straight into the Xcode playground.
As per the information above, we can see that our model is 75% accurate. If we want to improve the accuracy of the model, we can feed more training data. Once the desired level of the accuracy has been achieved we can save the model from Xcode playground. Just expand the image classifier area and save the model at the desired location. You can also add metadata to your model e.g licence, version, description etc.
Now that, we have fully trained machine learning model in the .mlmodel format that can be used with CoreML or other machine learning tools in the iOS apps.
Training ML Model with Swift (Automated)
The process of training machine learning models from Xcode playground is pretty easy and straightforward. However, if we want to repeat this task multiple time with different set of the data then it will soon become boring use Xcode playground. We can easily automate this swift script.
|
1 2 3 4 5 6 7 8 |
import Foundation import CreateML let training = URL(fileURLWithPath: "/tmp/ml/data-applenews/training") let testing = URL(fileURLWithPath: "/tmp/ml/data-applenews/training") let model = try MLImageClassifier(trainingData: .labeledDirectories(at: training)) let evaluation = model.evaluation(on: .labeledDirectories(at: testing)) try model.write(to: URL(fileURLWithPath: "/tmp/ml/data-applenews/applemews.mlmodel")) |
Save this file as ml.swift or whatever convenient name and replace the path with your training and testing data. We can run this script using the following command to automatically generate the machine learning models.
|
1 |
$ swift ml.swift |
At the end of running this script, you will see the model generated at the specified location.
Watch Demo Video
How To Use ML Models for iOS App Testing?
Now that, we have created a sample machine learning model using CreateML. The next thing is to explore how we can use this for the testing iOS apps. This is a really my area of research now and I would love to share my thought to the rest of the world how we can use these models for iOS app testing.
Visual Validation Testing with ImageClassifier
With the ImageClassifier, we can train machine learning models with images or screens of the iOS apps e.g home screen and other screens. If can get loads of images of the home screen with different data or dynamic data set of our iOS app we can train the model the which can identify the homepage of the app. Once we got the desired data, we can constantly test it with the new version of the homepage. We can still get the expected accuracy from the machine learning model then we are confident that our homepage UI is still OK and not has drastic changes or completely broken. In the Apple News app, we can use CreateML in following the way
- Get the loads images of the home screen on a different day and train the machine learning model with those images until you achieve the desired accuracy.
- Evaluate the model continuously with the latest homepage images
- Set the baseline for the model accuracy.
- When the model is model accuracy is steady we have confidence that home page is still working fine. If it drops down drastically then we can investigate the issues. e.g when the home page gets broken, the model accuracy will drop
This might be one of the use cases of using machine learning to test iOS apps.
Contract Testing of REST API with JSON Tabular Data
With CreateML, we can also train the tabular data in the form of CSV and JSON files. With loads of JSON responses from the API, we can feed the data to the machine learning model. We can test this model with the latest response from the API. If the model accuracy goes down then we can check that if the JSON response template has been changed or broken. This would be super useful for contract testing of the backend API
Challenges
The approach of using CreateML for iOS app testing will really work? I don’t have an answer to that yet. That’s the bigger question and its area of my research. I would love o hear your feedback, suggestions on this approach. However, in my brain, this approach is working very well but the challenges are
- How to collect a big amount of data?
- How to determine the collected data is sufficient?
- Where to host this huge amount of data securely?
- How t maintain the baseline data and newly added data for testing.
It’s definitely not easy but surely not impossible. It’s a matter of data if we can create a large amount of data to train machine learning model we can get to the point to perform solid tests on these data.
Resources
London Test Automation in DevOps Meetup Slides
Slides of my talk on CreateML for iOS APP Testing are available on SlideShare here
Slides from Merav Zaks are uploaded to Google Drive or Github
London TAD Meetup photos on the meetup website here
Official Apple: CreateML, CoreML, WWDC Video on CreateML
Sample Code
The code sample, assets of training/testing images, Xcode playground and slides for the meetup are available on Github here.
Conclusion
CreateML has made training machine learning models super easy. However, using those models for testing iOS apps would be interesting. In my brain, an idea of using CreateML for visual validation testing a contract testing for iOS app is working very well if we managed to get sufficient data to train ML models. In the time ahead, I will be doing more research in this area and might build some product around this area. Stay tuned with my blogs (XCBlog) and XCTEQ activities and product updates. If you want to share your ideas or help me in the research. Please get in touch. What do you think of this approach? What will be the pros and cons of using CreateML for testing apps? Please share in the comment below.