JavaScript Code Coverage with Istanbul

Tags:

Achieving a good code coverage is a useful practice in today’s software craftsmanship. For JavaScript applications, whether it is for the browser or for the server, many tools to check the statement coverage are available. What about branch coverage? Fortunately, such a tool is finally available: Istanbul.

Made by Krishnan Anantheswaran from the YUI team Yahoo! Cocktail team, Istanbul is extremely easy to use. It is suitable for both browser-based instrumentation or Node.js application analysis. Under the hood, Istanbul uses Esprima to parse JavaScript code and Escodegen to generate the instrumented version of the code. Istanbul is pure JavaScript, there is no native code or other kind of native libraries wrapper involved.

Istanbul requires Node.js, the package can be installed as:

npm install istanbul

Using Istanbul with command-line Node.js application is fairly straightforward. Assuming you have test.js like this:

x =42;
if(false)
     x =-1;

We can analyze the coverage by running:

istanbul cover test.js

which gives the following outcome:

=============================== Coverage summary ===============================
Statements   : 66.67% ( 2/3 )
Branches     : 50% ( 1/2 )
Functions    : 100% ( 0/0 )
Lines        : 66.67% ( 2/3 )
================================================================================

Even if you put all the above code in just one line, Istanbul can still give the correct coverage result. This is because Istanbul does not work based on line-by-line coverage, it does fully understands JavaScript syntax. Beside the quick text-based report, Istanbul also produces the coverage report in LCOV format. This way, the coverage report can be nicely shown, with color coding and other fancy features:

What about something more complicated? I wrote some time ago about the trap of checking only statement coverage, the example which demonstrated that is the following fragment. Now, if your unit test only verify the result of running inc(4) and you forgot you also need the check for inc(4,2), you won’t see the bug easily. This example is of course just an oversimplified example, in real-world application the problem is much harder to spotted.

function inc(p, q){
    if(q ==undefined) q =1;
    return p + q/q;
}

However, if we run Istanbul so that we can see the branch coverage, the report reveals the issue. Here the marker ‘E’ indicates that the else path of the branch is never executed and that should provoke a suspicion.

How does Istanbul work under the hood? It takes a JavaScript program and passes it through Esprima to get the syntax tree. Then, it injects some instrumentation by wrapping various syntax constructs. After that, a new JavaScript program is generated from the syntax tree by using Escodegen. When the program runs, the injected annotation is also executed accordingly and thereby the statistics of the program execution can be gathered.

While Istanbul has a mechanism to automate running Node.js application with instrumentation, nothing stops you from using the instrumented code in other JavaScript environments (such as web browsers). For this purpose, Istanbul has a special instrument command to perform only the instrumentation. It is up to you to process the coverage hooks and reports.

With Istanbul, tracking JavaScript code coverage has never been easier!

JavaScript Performance Analysis: Sampling, Tracing, and Timing

Tags:

Performance optimization of web applications is a hot topic these days. One of the related areas is of course optimizing the application code itself. For client-side application running in the web browser, this means speeding-up JavaScript code whenever possible. Premature optimization is not a good practice, it is crucial to locate which parts cause the slow down and need some improvement. Enter the logical first step of every optimization: initial performance analysis.

Sampling

Are you familiar with a JavaScript profiler? Many modern browsers already include a very useful profiler to look for JavaScript performance problem. The workflow is like this. First, you press a button to start the full performance monitoring of your application. After the application runs for a while, you can press another button to stop the monitoring. The outcome is something top-down like the following screenshot, the breakdown gives a very useful information on where your application spends the most time.

Under the hood, activating the monitoring causes the JavaScript engine to kick an sampling profiler. It is named that way because the profiler will look at the state of the virtual machine at a predefined interval, e.g. 1 ms in case of V8. The important state information will be collected and later it will be used to construct the profiling view as depicted in the above screenshot.

When carrying out a profiling like this, be advised of the observer effect. The extra monitoring adds a certain overhead to the code execution and hence, the actual timing would be different to the case where there is no instrumentation at all. The difference is usually minimal, in particular since the entire application is affected. Yet, take that into account as you move forward to make any conclusion.

Tracing

While timing information obtained from the profiler is useful, sometimes you are also interested not in how fast a certain operation is carried out, but also what happens during that time. This is like doing an X-ray on your program execution. Fortunately most modern browsers support the console API console.trace will give you exactly that information:

Another way to gather the traces is by instrumenting the JavaScript source. This is a technique I’ve used to find out the exact execution sequence when an application starts. For example, in that experiment I found out that a simple jQuery Mobile site will invoke over 4000 function calls. Note that the number of calls itself does not tell you much. However, tracking this over time, or even for every check in, can be really helpful. For example, if suddenly someone commits a bug fix which brings the function calls to 8000, this should trigger a red flag. That can be one of your protective multilayer defense.

Scalability is another excellent area of application tracing. If you have an address book application, sorting the contacts the alphabetically can be really fast if you only have 10 entries. However, here you are not interested in the absolute time of the sorting. You also want to know how it handles the address book with 100 entries, 1000 entries, and so on.

Formal analysis of the complexity can be complicated or prohibitively expensive. This is where empirical run-time analysis kicks in. For example, you can instrument Array#swap and plot the number of function calls vs address book entries. If your team member implemented it using bubble sort instead of something faster like binary sort, that chart will reveal it.

Timing

After you locate the problematic spot in the performance, the next step is obviously to fix it. In many cases, speeding up some parts of the application is not really difficult. In other cases, you have to try different strategies and see which one fits the performance criteria. Often, it is as simple as figuring out which implementation is the fastest. This is where timing the execution of a function is useful.

Accurate timing is far from trivial, this has been covered in many articles, e.g. Bulletproof JavaScript benchmarks. Unless you have a lot of time to learn statistics and uncover cross-browser secrets, it is easier to use a ready-made benchmark library such as Benchmark.js. For a quick comparison, using the popular JSPerf (which uses Benchmark.js under the hood) is highly recommended. You would also have the chance to try it on different browsers and devices, just to ensure that your strategy is not biased towards a particular implementation.

Beside careful timing, it is also important to pay attention to the benchmark corpus. Whenever possible, choose something which resembles its real-world usage. For example, if you try different ways to sort the contacts in the address book application, make sure you supply a representative list for the benchmarks, more than just a useless array of ['a', 'b', 'c'].

Last but not least, remember that optimization is not the destination, it is a journey!

Quality Code via Multiple Layers of Defense

Tags:

Have you written a book, a dissertation, or a report? If you prepare well, everything should be almost perfect until you need to submit it to the editor, the professor, or the supervisor. In fact, almost likely you will read the manuscript again and again, just to ensure that there is no silly mistake, from typos to missing figures, which can be spotted easily in the draft. Before the document is going through another stage of proof-reading, you want to know as early as possible if something needs some fixes.

Fortunately, the same “multiple layers of defense” strategy applies to modern software development. For example, even though the compiler will scream at you if your code does not make sense, there is no reason to skip fixing it, if you happen to see it on your editor. Many complex application requires a rigorous testing procedure, yet it does not harm anyone to run the application through a fast smoke testing first. In fact, by the use of git pre-commit hook, you can run the smoke test before the defective code gets landed in the repository.

The principles are summarized as:

  • If you screw up, you want to know earlier than later
  • The first layer should have the lowest cost and effort

For client-side web development, this approach can be a time-saving. When you are in the coding stage, ideally the editor or the IDE which you use would give an instant feedback whenever there is something suspicious. It does not need to catch all possible cases, it has to be just good enough to filter the most common mistakes. If you leak a variable and it pollutes the global, there is no need to launch the unit tests before it gets caught.

Using CoffeeScript/Dart/TypeScript? Verify the compiled JavaScript with a syntax validator (because those code generators sometime have bugs, too). If you don’t use strict mode properly, there is no reason to wait for the full-blown QA workflow which involve a dozen web browsers (possibly even cloud-based) to kick in and then tell you that something is wrong. If your template handling does not produce the right DOM element, there is no use of running the functional tests on a whole set of mobile devices.

Just like a bodyguard, your layers of protection can be very subjective. Newbies may want more aggressiveness so that they can reduce the mistake probability. A typical configuration of those layers may look like the following:

This also demonstrates that an overlapping functionality does not always mean a substitution. Many tools complement each other quite well. Just because you have 100% code coverage does not mean you can’t benefit from a blazing-fast syntax error checking. Even if you aim to support legacy browsers, it does not mean you exclude yourself from doing headless testing.

Everyone makes mistakes. Fix them quickly and move on.

IE 10 Web Performance Quick Check

Tags:

Microsoft finally made Internet Explorer 10 available for Windows 7. It is still in the preview stage, however I’m already curious about its performance characteristics. I decided to run some quick check to see how it stacks up. The test machine is an average Toshiba laptop (Core i3, 2 GHZ) running a fully up-to-date Windows 7 Home Premium (with SP1).

Like what I did with Nexus 7, other tablets, and various smartphones, my first interest is (always) in the DOM performance. Many rich/interactive sites and single-page web application still benefits a lot from a speedy DOM manipulation. I ran the quick Dromaeo DOM core tests and here is the result (longer is better). Firefox is the undisputed, Internet Explorer is unfortunately not in the same ballpark.

Microsoft always claims that the graphics performance of Internet Explorer is really good. Fortunately for us, this is not a baseless claim at all. Running my underwater demo, which stresses the speed of pixel manipulation in HTML5 Canvas, proves that the statement holds water. The chart follows.

Some time ago Mozilla released Kraken, the benchmark suite geared towards the representation of code patterns found in real-world application (shorter is better). Firefox and Chrome dominate, Internet Explorer is twice as slow (although it manages to beat Opera).

For pure JavaScript performance, many others have tested various Windows browsers with SunSpider, V8, or Octane benchmarks. I opted for something different, i.e. running a parser written in JavaScript. This is easy via the online Esprima benchmark. It exercises string processing, object construction, and recursive functions call. The chart below demonstrates that Internet Explorer 10 still needs to catch up.

Windows 7 is still used by millions desktops users, having many choices of fast browsers would be delightful to those users. Hopefully Microsoft would address some of these performance issues in the final release of Internet Explorer 10.

Polluting and Unused JavaScript Variables

Tags:

There are two common mistakes when handling JavaScript variables, accidentally making a variable global and declaring a variable which is never used anywhere. Many well-known code linters can warn the developers when those cases are observed. How does the detection process actually work?

(The leakage sign was made by Robert Ingil)

Global Pollution

Let us see the first one, the case of global pollution. A code fragment which demonstrates the issue follows:

var height;
// some fancy processing
heigth = 200;

Due to a spelling mistake, the value is assigned to some variable different that what it is declared in the first line. Checking for this case is rather easy. As long as we get the AST (abstract syntax tree) of the code, we can look for such an assignment. The syntax tree itself is relatively trivial to obtain, we just pass the code to a parser like Esprima. Walking the syntax tree, we keep track of variable declaration so that we can cross-reference it within any assignment (of course, we must take into account complications such as variable hoisting and function scope). In fact, there is a tool that does these verification steps, it’s called leaky, made by David Björklund. It must be used with Node.js and the package can be installed easily with

npm install leaky

The above mentioned logic, along with the syntax tree traversal, is less than 100 lines. Running leaky with the previous example code gives this output:

test.js:3
heigth = 200;
^
LeakError: global leak detected: heigth

If you develop an application or a library, this leak detection is suitable to be inserted in your testing workflow. A non-zero exit code is used to indicate that there is something wrong. Thus, the test section in your package.json could be something like:

"scripts": {
  "test": "leaky mycode.js && node run-tests.js"
}

If you update your code and it unintentionally leaks some variables, your test won’t even run as leaky will block further steps until the situation is resolved. Obviously, you could also plug the check as part of your git pre-commit hook.

Now, there can be few corner cases of global polluting which won’t be detected purely by static analysis. In that case, it is suggested to use strict mode so that such a pollution will be caught at run-time.

Unused Variables

For the second case, declaring a variable which is never used anywhere, the principle is surprisingly quite similar. Here we need to do the opposite because a variable needs to be marked as used if it is encountered in an expression. While we still need to traverse the syntax tree, it will require more steps as every possible usage of a variable, from the loop initializer to postfix operation, needs to be verified. All in all, the implementation won’t exceed 200 lines as demonstrated by the unused project from Tomaz Muraus and Roman Shtylman. Again, Node.js users can get the package easily via:

npm install unused

Running it on the previous code fragment shows before gives:

test.js
     height - on line 1

Most certainly, this tool can be used on a much more complicated code. In addition, an unused function argument can be valid, in particular when the function is part of the callback. For example, see the following block:

[4, 5, 6].forEach(function (value, index) {
    console.log(index);
});

In this case, value will be correctly reported as being unused. However, it is perfectly valid to keep it in that construct. Fortunately, unused can be invoked with an additional command-line parameter which would ignore specific argument (place)holders.

Just like leaky, it is perfectly reasonable to include unused as part of your development workflow, either within the test procedure or the pre-commit hook, or even integrating it as your editor’s behavior.

Try leaky and unused on your favorite application code and see if they show something interesting!

Checklist vs Mission

Tags:

Imagine the following situation. Your little brother starts to smoke cigarettes and your parents are worried about that. As you are close to him, they ask you for your help. The goal is to have your brother stop smoking. What would you do?

A straightforward, pure logical way of convincing your poor brother to avoid smoking is by collecting a stack of medical research paper containing the encyclopedic proof of all negative effect of cigarettes. It is very unlikely that this strategy will be successful. And if it does not work, your parent won’t be happy about it. However, you can shrug off and walk away, claiming that you have done your part. This is the exact mentality if you only want to get rid of the burden you bear. Your objective from the very beginning is sadly only to mark an item in the checklist.

Things will be different if you are committed to a mission. Now the game play is different. You will do an extensive research on why things happened in a certain way. You will ask around, perhaps checking some of your brother’s close circle, and collect a better understanding of the sudden behavior change. You are doing your best to maximize your chance of success, regardless whether you can achieve the ultimate success or not (some things are beyond your power).

The same philosophy applies to software development as well. A sprint can fail miserably if everyone carries out a task so that he can have something to say in the next day’s scrum meeting, the goal is shifted from “creating a solid product” to “working on FooBar so I won’t get fired”. A testing workflow can be very ineffective if every quality engineer looks forward to fill the test worksheet as much as he can, the objective is no more “ensuring high-quality product” as it is merely “checking all the marks in the plan”. An engineering organization can disintegrate quickly once each VP only optimizes for the annual target, the responsibility gets degraded from “promoting a great team” to “making hay while the sun shines”.

Next time you want to respond to an idea, to propose a new change, to share an interesting development, or just to defend your own opinion, think about it again: is this part of your (life) mission or is it only to cross an entry from your checklist?