Automatic visual diffing with Puppeteer

So testing, right? We should do it. The thing is, testing is hard, and good testing is reaaaaaaally hard, and tbh Iā€™m pretty bad at testing. So I end up not testing my apps, and then I feel guilty about it, but Iā€™ll stop you now: you canā€™t run guilt on Travis. If this sounds familiar, then this blog post is for you.

I did a little song-and-dance that sets up Puppeteer* , takes screenshots of your app (like, all the routes you care about), and then compares them to the ā€œgoldenā€ ones. If they match, your test passes! Yes, it only works on Chrome. No, itā€™s not actually unit testing. Yes, itā€™s literally just counting pixels but you know what? It counts them in both a wide and a narrow viewport size and any testing is better than no testing at all; fight me.

* Puppeteer is an npm library that lets you control Chrome. You know, like a puppet. In particular, Puppeteer makes it super easy to take screenshots (and click on things in your page). Itā€™s like a waaaaaaay less infuriating Selenium, but infinitely harder to spell.

This post looks long because Iā€™ve put all the code I have so that you can copy paste it. Skip to the good part if you already know how to test.

Do the npm

If you want to test things with Puppeteer, you have to setup a thing for the tests, a server that launches your site, and then Puppeteer to look at that site. I have this in my package.json to wrangle these things:

"devDependencies": {
  "chai": "^4.1.2",
  "mocha": "^5.0.0",
  "puppeteer": "^1.0.0",
  "pixelmatch": "^4.0.2",
  "polyserve": "^0.23.0"
}

Explanation:

Set up your test

In order to tell Puppeteer to investigate your site, you need to:

  1. start a test suite
  2. that sets up a local server
  3. and in each test tells Puppeteer to do something.

My setup looks like this:

const puppeteer = require('puppeteer');
const expect = require('chai').expect;
const {startServer} = require('polyserve');

describe('šŸ‘€ screenshots are correct', function() {
  let polyserve, browser, page;

  // This is ran when the suite starts up.
  before(async function() {
    // This is where you would substitute your python or Express server or whatever.
    polyserve = await startServer({port:4000});

    // Create the test directory if needed. This and the goldenDir
    // variables are global somewhere.
    if (!fs.existsSync(testDir)) fs.mkdirSync(testDir);

    // And its wide screen/small screen subdirectories.
    if (!fs.existsSync(`${testDir}/wide`)) fs.mkdirSync(`${testDir}/wide`);
    if (!fs.existsSync(`${testDir}/narrow`)) fs.mkdirSync(`${testDir}/narrow`);
  });

  // This is ran when the suite is done. Stop your server here.
  after((done) => polyserve.close(done));

  // This is ran before every test. It's where you start a clean browser.
  beforeEach(async function() {
    browser = await puppeteer.launch();
    page = await browser.newPage();
  });

  // This is ran after every test; clean up after your browser.
  afterEach(() => browser.close());

  // Wide screen tests!
  describe('wide screen', function() {
    beforeEach(async function() {
      return page.setViewport({width: 800, height: 600});
    });
    it('/view1', async function() {
      return takeAndCompareScreenshot(page, 'view1', 'wide');
    });
    // And your other routes, 404, etc.
  });

  // Narrow screen tests are the same, but with a different viewport.
  describe('narrow screen', function() {
    beforeEach(async function() {
      return page.setViewport({width: 375, height: 667});
    });
    it('/view1', async function() {
      return takeAndCompareScreenshot(page, 'view1', 'narrow');
    });
    // And your other routes, 404, etc.
  });
});

You can test all sort of things here, by the way. Puppeteer lets you interact with the page (click on buttons, links, etc), so maybe you want to trigger different UI states before you screenshot them (like narrow view but also with the navigation drawer opened).

Filling in the blanks

All the heavy lifting (which isnā€™t very heavy tbh) is done in takeAndCompareScreenshot:

// - page is a reference to the Puppeteer page.
// - route is the path you're loading, which I'm using to name the file.
// - filePrefix is either "wide" or "narrow", since I'm automatically testing both.
async function takeAndCompareScreenshot(page, route, filePrefix) {
  // If you didn't specify a file, use the name of the route.
  let fileName = filePrefix + '/' + (route ? route : 'index');

  // Start the browser, go to that page, and take a screenshot.
  await page.goto(`http://127.0.0.1:4000/${route}`);
  await page.screenshot({path: `${testDir}/${fileName}.png`});

  // Test to see if it's right.
  return compareScreenshots(fileName);
}

Getting the golden screenshots

This bit is easy. Make a different test suite (just make sure you donā€™t run it every time you run your tests), and run the page.goto and page.screenshot lines for all the routes youā€™re testing. I recommend doing the viewport dance too, to get both the wide and narrow screen ones for freeeeee (I am using just the viewport size here, because thatā€™s how my app works. Puppeteer lets yo do device emulation and all sorts of other goodness, so just read the docs). Put all these screenshots in a place; I put mine in a folder called test/screenshots-golden/.

The thing that does the diffing

This is the logic in compareScreenshots, and itā€™s basically straight out of the Pixelmatch docs:

function compareScreenshots(fileName) {
  return new Promise((resolve, reject) => {
    const img1 = fs.createReadStream(`${testDir}/${fileName}.png`).pipe(new PNG()).on('parsed', doneReading);
    const img2 = fs.createReadStream(`${goldenDir}/${fileName}.png`).pipe(new PNG()).on('parsed', doneReading);

    let filesRead = 0;
    function doneReading() {
      // Wait until both files are read.
      if (++filesRead < 2) return;

      // The files should be the same size.
      expect(img1.width, 'image widths are the same').equal(img2.width);
      expect(img1.height, 'image heights are the same').equal(img2.height);

      // Do the visual diff.
      const diff = new PNG({width: img1.width, height: img2.height});
      const numDiffPixels = pixelmatch(
          img1.data, img2.data, diff.data, img1.width, img1.height,
          {threshold: 0.1});

      // The files should look the same.
      expect(numDiffPixels, 'number of different pixels').equal(0);
      resolve();
    }
  });
}

šŸ’Æ Itā€™s all worth it

Now, when you run your tests (mocha test/ --timeout 5000 in my case), you get something like this:

10/10 passing tests

And if it fails, you get an error and the number of pixels youā€™re off by.

ā­ļø

Now go on, navigate to all your routes and test your stuff, and thank me with a photo of your dog.

« 2017: another year in review An intro to Reinforcement Learning (with otters) »