Visual Regression

In recent days, I came across mind blowing topic called ‘Visual Regression‘. There is still no proper definition of the visual regression but idea is fairly simple. In the Visual Regression world, we take screenshots of the live applications as a reference of base screenshots, and while we develop additional features we compare new screenshots with the reference or base screenshots to make sure existing functionality is not broken. If screenshots differ then visual regression testing tools warn you about the changes. There will be more in depth discussion in the #LDNSE 12 London Selenium Meetup at Huddle. Keep an eye on the meetup ‘Visual Regression : PhantomCSS and Selenium“. Details here

Visual regression testing sometimes called ‘CSS Regression Testing. There are many tools available now to perform visual regression and it’s challenging to pick the correct tool based on your project need.  Visual regression tools works very well to test web and responsive layouts. The visual regression testing tools mainly uses following libraries to capture screenshots

• Selenium
• PhantomJS
• CasperJS

Visual Regression Options

Currently there are lots of options to perform CSS regression. Let’s have brief look at some of the popular visual regression tools

• PhantomCSS

PhantomCSS is a very popular CSS regression testing framework developed at @Huddle by James Cryer. PhantomCSS can be scripted with CasperJS.

• BackstopJS

BackstopJS is visual regression tool developed by Garris. You can find step by step tutorial of BackstopJS on CSS Tricks site but since them there are many updates in the BackstopJS so we will cover latest setup in this post.

• Wraith

Wraith is another screenshot comparison tool developed by the developers at BBC News. The detailed documentation of the Wraith found here

• Success

Success is another nodeJS package can be used for the CSS regression testing. Now we will discuss and compare each tools but in this post we will discuss how to setup BackstopJS from Scratch.

BackstopJS

Although detailed step by step tutorial on BackstopJS on CSS-Tricks website, there are few changes since then as Bower dependencies are deprecated.

Installation

Now that, we assume that we have NodeJS installed. Now create  new directory ‘visual-regression-backstopjs’

Now, let’s create a package.json file with all our dependencies for the BackstopJS

Let’s install all these packages

This will create ‘node_modules/backstopjs’ directory. Now cd into that directory and install node dependencies

 Generate Config file

Now we can generate backstopjs.json config file.

This will create ‘backstop.json’ file in the project We can modify this accordingly. It will look like this

Generate Baseline Screenshots

Now let’s create baseline images by running

This will generate base images to compare against. The images will be in the ‘node_modules/backstopjs/bitmaps_reference’ directory

Generate Test Screenshots

Now that, we have generated reference screenshots to compare against, let’s generate new screenshot and compare both

This will test latest screenshots against the refence screnshots.The newly created images will be in the ‘node_modules/backstopjs/bitmaps_test’ directory

Once it finishes test. Google chrome will pop up with result.

When it fails, it shows the screenshot diff.

GitHub Repo  : visual-regresion-backstopjs

https://github.com/Shashikant86/visual-regression-backstopjs

 

This way we can use backstopJS as a visual regression testing tool.