.

Tags:

For modern web application development, having dozens of unit tests is not enough anymore. The actual code coverage of those tests would reveal if the application is thoroughly stressed or not. For tests written using the famous Jasmine test library, an easy way to have the coverage report is via Istanbul and Karma.

For this example, let’s assume that we have a simple library sqrt.js which contains an alternative implementation of Math.sqrt. Note also how it will throw an exception instead of returning NaN for an invalid input.

var My = {
  sqrt: function(x) {
    if (x < 0) throw new Error("sqrt can't work on negative number");
      return Math.exp(Math.log(x)/2);
  }
};

Using Jasmine placed under test/lib/jasmine-1.3.1, we can craft a test runner that includes the following spec:

describe("sqrt", function() {
  it("should compute the square root of 4 as 2", function() {
    expect(My.sqrt(4)).toEqual(2);
  });
});

Opening the spec runner in a web browser will give the expected outcome:

jasmine_runner

So far so good. Now let's see how the code coverage of our test setup can be measured.

The first order of business is to install Karma. If you are not familiar with Karma, it is basically a test runner which can launch and connect to a specific set of web browsers, run your tests, and then gather the report. Using Node.js, what we need to do is:

npm install karma karma-coverage

Before launching Karma, we need to specify its configuration. It could be as simple as the following my.conf.js (most entries are self-explained). Note that the tests are executed using PhantomJS for simplicity, it is however quite trivial to add other web browsers such as Chrome and Firefox.

module.exports = function(config) {
  config.set({
    basePath: '',
    frameworks: ['jasmine'],
    files: [
      '*.js',
      'test/spec/*.js'
    ],
    browsers: ['PhantomJS'],
    singleRun: true,
    reporters: ['progress', 'coverage'],
    preprocessors: { '*.js': ['coverage'] }
  });
};

Running the tests, as well as performing code coverage at the same time, can be triggered via:

node_modules/.bin/karma start my.conf.js

which will dump the output like:

INFO [karma]: Karma v0.10.2 server started at http://localhost:9876/
INFO [launcher]: Starting browser PhantomJS
INFO [PhantomJS 1.9.2 (Linux)]: Connected on socket N9nDnhJ0Np92NTSPGx-X
PhantomJS 1.9.2 (Linux): Executed 1 of 1 SUCCESS (0.029 secs / 0.003 secs)

As expected (from the previous manual invocation of the spec runner), the test passed just fine. However, the most particular interesting piece here is the code coverage report, it is stored (in the default location) under the subdirectory coverage. Open the report in your favorite browser and there you'll find the coverage analysis report.

branch_uncovered

Behind the scene, Karma is using Istanbul, a comprehensive JavaScript code coverage tool (read also my previous blog post on JavaScript Code Coverage with Istanbul). Istanbul parses the source file, in this example sqrt.js, using Esprima and then adds some extra instrumentation which will be used to gather the execution statistics. The above report that you see is one of the possible outputs, Istanbul can also generate LCOV report which is suitable for many continuous integration systems (Jenkins, TeamCity, etc). An extensive analysis of the coverage data should also prevent any future coverage regression, check out my other post Hard Thresholds on JavaScript Code Coverage.

One important thing about code coverage is branch coverage. If you pay attention carefully, our test above is still not exercising the situation where the input to My.sqrt is negative. There is a big "I" marking in the third-line of the code, this is Istanbul telling us that the if branch is not taken at all (for the else branch, it will be an "E" marker). Once this missing branch is noticed, improving the situation is as easy as adding one more test to the spec:

it("should throw an exception if given a negative number", function() {
  expect(function(){ My.sqrt(-1); }).
    toThrow(new Error("sqrt can't work on negative number"));
});

Once the test is executed again, the code coverage report looks way better and everyone is happy.

full_coverage

If you have some difficulties following the above step-by-step instructions, take a look at a Git repository I have prepared: github.com/ariya/coverage-jasmine-istanbul-karma. Feel free to play with it and customize it to suit your workflow!

  • JT

    Great post as always. Thank you!

    • http://ariya.ofilabs.com/ Ariya Hidayat

      Glad to know it’s useful.

  • Citroenboom

    Totally out of the scope of this blog post, but I have a question: the main reason I haven’t started using code coverage tools such as instanbul, is that I’m anxious I will overtest stuff just to make the code coverage better. What is an acceptable figure?

    When I started out with TDD, I was testing way too many things, and I learned just to test the important stuff, not the whole project in every minor detail. What are your thoughts?

  • Matthew Kastor

    Nice. Thanks for sharing this.

    • http://ariya.ofilabs.com/ Ariya Hidayat

      You are welcome.

  • Aats

    Hey Aria,
    Thanks for blogging on Code coverage for JS Unit tests.
    I’ve been using Karma quite a lot for my Unit tests and I get all the nice things around coverage (with its inbuilt support with Istanbul). Everything works great for me in the Unit Test world.

    However, I also have functional tests in which I don’t load/serve the js source code via Karma. Instead, only tests are served via Karma and those tests get the js source code from a different server. The interesting thing is that different server is running on the same machine as Karma server is running on.

    Any idea, if there is a way to get code coverage on my functional tests?

    • http://ariya.ofilabs.com/ Ariya Hidayat

      Check istanbul middleware project which can serve instrumented JavaScript code for your testing. Instead of the plain vanilla code, your test will load the instrumented version and thus you can gather the coverage instrumentation yourself.