one year of wandering headlessly

January 23rd, 2012 Tags: , , ,


One year ago I decided to bite the bullet and released PhantomJS to the world. It was the opening move in my own initiative to curate at least three open-source projects. While it was far from being really really ready, I was very content that I didn’t postpone it much further. After all, isn’t worse actually better?

After five stable releases, from the initial 1.0 to the latest Glory of the Snow (all named after flowers, read the full story), PhantomJS gets stronger and gains more traction and momentum. Let’s have a quick review on how people use it.

Since it is essentially a headless version of WebKit, web page automation is the main use cases. For scraping and navigation, CasperJS (which is using PhantomJS) from Nicolas Perriault strikes as an excellent example on how to do this. Check its documentation and examples, most of your automation needs will be probably covered. For automated page analysis, see also James Pearce’s confess.js which you can use (among others) to analyze page speed and cache manifest usage.

Automatic manipulation of browser content is the key to application testing. PhantomJS slowly becomes a common way to run the tests headless in many test frameworks. It is really easy to hook PhantomJS to the testing infrastructure, there are reports people running it with Jasmine, QUnit, FuncUnit, etc. Of course that once it is done, making it part of the workflow with various continuous integration system, anything from TeamCity and Jenkins is rather fun.

Speaking about testing, Capybara is often the choice for Rails/Rack developers. Thanks to Jon Leighton, finally we have Poltergeist, a solid PhantomJS driver for Capybara. While it does not substitute multi-browser testings via Selenium/Web Driver of various real-world browsers, this driver facilitates very fast smoke testing, just to ensure that you don’t do any stupid mistake before launching the extensive browser testing.

In addition, again thanks to Jon’s effort and the cool guys at Travis, PhantomJS is supported as part of your testing for Travis CI. There are already projects which leverage this (again as part of quick automated smoke tests), among others CasperJS and Ember.js (formerly known as SproutCore 2.0).

If you are not still convinced, there is a bunch of companies out there who use PhantomJS internally as part of the application development, from Sencha, Torbit, Betfair, to Time Warner Cable. Last year alone there were more than 80 articles and blog posts showing different use cases. Also, we even have leading company such as KDAB which put engineering effort into improving PhantomJS. And while I can’t reveal the details yet, expect to see more support from commercial organizations to the future development of PhantomJS this year.

The rather simplistic official site phantomjs.org successfully gets more than 120 thousands visits. Thanks to Maurice Svay, the site will sport a fresh redesign very soon. In the mean time, the project page hosted at Google Code easily hits half million views from more than 20 thousands unique visitors. The community discussion happens mainly in the mailing-list which currently grows to more than 300 members.

Technically, PhantomJS also serves as my playground for some of the software development best practices I experiment lately, everything from predictable time-based release schedules to extensive uses of issue tracker (read about it in the previous blog post). If you understand the concept of lean startup, these strategies may sound familiar.

It’s been an extremely fun ride. Thank you PhantomJS users and contributors for the amazing one year. We will not stop here for sure. Welcome to the year of headless dragon!

Read More »

senchacon 2011 videos

January 20th, 2012 Tags: , , , , , ,

What happens in Austin does not stay in Austin!

As I already wrote before, at the last SenchaCon 2011 in Austin, TX there were two presentations that Jarred and I delivered. The first one was about a high-level overview of WebKit, how to leverage WebKit for application testing and optimization. The second one discussed the machinery behind hardware acceleration on mobile browsers. The slide decks for these two talks have been made available recently.

Here comes the good news. Now you can also enjoy the entire SenchaCon tracks since the recorded videos can be watched at your convenience. For these two talks I mentioned, here are the complete goodies:

If you are interested in similar topics, check also other related talks I’ve involved with in the past.

Enjoy!

Read More »

books 2011

January 16th, 2012 Tags: ,


I always tell myself to read more books. Somehow last year wasn’t too much of a success. Here are some books (not necessarily recently published) I managed to read in 2011, listed in no particular order, which I would not hesitate to recommend to others.

non-fiction

  • Hyperspace by Michio Kaku
  • Linchpin by Seth Godin and Hugh Macleod
  • Coders at Work by Peter Seibel
  • Founders at Work by Jessica Livingston
  • Drive by Daniel Pink
  • Winning by Jack Welch and Suzy Welch
  • I’m Feeling Lucky by Douglas Edwards
  • JavaScript Patterns by Stoyan Stefanov
  • The Way of the World by Ron Suskind

fiction

  • Abraham Lincoln: Vampire Hunter by Seth Grahame-Smith
  • The Fist of God by Frederick Forsyth
  • Force of Eagles by Richard Herman Jr.
  • Spy Hook by Len Deighton
  • His Last Bow by Arthur Conan Doyle
Read More »

small-scale software craftsmanship

January 12th, 2012 Tags: , , , ,

Photo by Trey Ratcliff (license: Noncommercial Creative Commons).

Building software can be as tricky as managing a complicated metropolitan city. Yet, as with many other things related to delicate handling, there are always approaches which are, time and time again, proven to streamline the workflow and increase the chance of success.

During my amateur and professional time doing software development, I collected selection of best development practices which I had hoped I could really exercise. With PhantomJS and Esprima, I finally got the chance to give these practices a try. Often they work, sometimes they don’t.

Note that obviously these are not all what we are doing. There are the typical mandatory activities like test-driven approach, extensive code review, proper repository access, agile trick, and the usual yada yada.

plan your release like you plan your wedding

When you say, “Sweetheart, we’ll have our wedding on the 4th of July this year”, you mean it. You won’t come back again in several weeks and then express “I’m not ready yet, let’s postpone it by few months”. Barring any extraordinary events (earthquake, nuclear holocaust, or world war), you give your best to fulfill your commitment. You want badly that you two get married on that date.

The implication of this is huge: you give your words to the users that you will ship a release and you won’t slip at all cost. It helps if you put a sticky note with that fundamental premise in front of your desk. Read it every morning, loud if necessary.

Never forget that real artists ship.

predictable time-based release

Now that you set a contract between you and your users about the release, pick a reasonable date and stick with it. I was always undecided about time-based releases but over the last few years I can see how this makes sense after all. Engineers are excited about cool new stuff, customers are thrilled about getting the best possible values. The first group is the early adopter, but the folks who stick with your product always need something stable to serve their own customers.

There is a dramatic consequence of this approach with respect to feature development. First of all, you need to align your development timeframe with the release plan. This is always good as not everyone can afford “release it when it’s done” strategy. But the best side effect that you want to isolate your refactoring, rewrite, redesign, and the other two dozens potential screw up. Modern version control, like Git, makes it easy to create a special feature branch, while at the same time also doing frequent merge and QA from the stable branch.

A small choice I make for PhantomJS, it’s always released every season. To make it more fun, the release date is chosen to be a solstice or an equinox. Astronomy does not need to be boring, right?

use the project tracker extensively

Even if you are the only developer working on your project, there is a chance that you don’t remember everything. Assume right now you still remember most of the important stuff, in few months things will look different. Real life takes over sooner or later.  Out of sudden you have a baby and good luck trying to recall how you ended up with that obscure design choice.

A lot of projects use some sort of tracking software, usually web-based, which allows you to create an entry and track them. For software hosted at Google Code, Github, SourceForge, etc, the tracker software is pretty good. Use the tracker for note taking. Capture your Eureka moment and save it for your future kids. It is imperative that every little decision must be properly documented (I know, it’s just common sense!). For large-scale projects (like WebKit), this is logical but I believe any small-scale projects should do the same.

If you’re still not convinced, remember that commit log is frozen and read-only. If you commit a bug fix and 42 days later you realize it was a mistake, you can’t change the old commit log to say “oops”. However, if someone tracks some regression and finds that commit (via git blame, for example), then the commit log does not represent the current amended situation. Now, if there is an issue linked from the commit log, it’s easy to follow it and possible find the reopened bug, along with further investigation result. This also helps if the commit message blames/credits a wrong person or refers/excludes a different product.

One monumental benefit of recording everything in the tracker: it is searchable by anyone, including (of course) the future version of you. Now go one step further and mandate that the link to the issue is mentioned in the commit log. A tracking system usually has some sort of hooks so that it can automatically even link back to the commit or do other related activites (closing the bug upon a certain keyword), if the commit log mentions the issue.

This cross referencing is extremely time-saving in the long run. One day you will decide to run git blame, find the culprit check in, and thus can easily track the real issue. Perhaps even the other way around, a new bug is reopened and you can trace back the original changes related to the bug. On the DRY front, such a practice centralizes all the info related to future plan, defects, known problems, etc in the most logical place: the issue tracker itself.

Last but not least, issues aggregated based on the targeted version can easily serve as your mission center, a quick overview of the status and the plan. For example, Esprima has this wonderful grid view which shows its status with regards to the next release, a kind of a project dashboard.

semi-esoteric code names

Why? Mostly because it is fun and it gives a sense of personality. Your software is not just a series of ones and zeros packaged in a nicely presented user interface, it actually has a lot of stories, dramas, and tragedies behind it. After all, I’d like to refer the capital of my country with its name, Jakarta, rather than 6°12′S 106°48′E.

Look at how some browers are named (exploration theme): Navigator, Explorer, Konqueror, Safari. Perhaps you also see the connection in Android release names like Eclair, Froyo, and Gingerbread (beside the fact that they are tasty)? Are you aware that Maemo versions (Chinook, Diablo, Fremantle, and Harmattan) are named after winds, possibly a way for Nokia to indicate "the wind of change"? For some reasons, Mozilla JavaScript engine has a lot to do with a specific primate: SpiderMonkey, TraceMonkey, IonMonkey. Ubuntu is well-known for its funky codenames like Natty Narwhal and Maverick Meerkat.

For PhantomJS I pick up the name of specific flower related to the season of the release: Cherry Blossom, Birds of Paradise, Water Lily, Glory of the Snow. There is a whole wiki page on the story behind the names, if you care.

Any other uncommon best practices for small-scale software you always practice?

Read More »

api shame: non-descriptive property names

January 6th, 2012 Tags: , , ,

I had a lot of fun writing about the Boolean trap issue. I also have more similar API-related discussions in the pipeline. Rather than writing a long-ish essay again, I decide to put out more bite-size chunks. Here is the first.

Careful when choosing the name of an object property!

Let’s say you create a date picker component, something like depicted in the following screenshot:

To set the range of the year where the user is allowed to choose a certain date, here is a bad examples of the property names:

picker.yearFrom = 1900;
picker.yearTo = 2100;

It’s quite “conversational” to decide the names like yearFrom and yearTo. A much better choice would be like this:

picker.minimumYear = 1900;
picker.maximumYear = 2100;

You could use other idioms, like start and end, if you like. The philosophy is the same, when you define a property, try to say it loud in a normal English sentence.

Here is another example. Say you want to introduce yourself:

“Hi. Call me Adam.”

Does that mean you are going to write code like this?

person.callMe = 'Adam';

Obviously not. Imagine your introduction has been changed to:

“Hi. My name is Adam.”

which is analog to the following:

person.name = 'Adam';

as if you say it (loud):

The name of the person is Adam.

Let’s for the moment assume that this introduction happens in an airport, as you exchange greetings with fellow passengers in the waiting room. The conversation continues:

“My flight is from SFO to JFK.”

Let’s write the code for that. Maybe these lines?

flight.from = 'SFO';
flight.to = 'JFK';

Again, it’s very conversational. Technically no grammar is violated is your original sentence, though imagine if it would have been rewritten as:

flight.departure = 'SFO';
flight.destination = 'JFK';

Less ambiguous? I bet!

In many cases, it’s impossible to choose one-word property name. In fact, it’s better to avoid it if that may cause some confusion.

If you have a scrolling list, where the user can swipe his finger horizontally or vertically, one way to specify it is:

X.scroll = 'horizontal';
Y.scroll = 'vertical';

Still considering the original observation that:

the code is usually written once but read many times.

those two lines lead to an ambiguity because scroll as a word can mean a lot. At least, it’s a verb and also a noun (though the latter often connects to the magical world). What you really want people to understand is perhaps:

X.scrollDirection = 'horizontal';
Y.scrollDirection = 'vertical';

Say it loud and it makes sense:

The scroll direction of X is horizontal.

I hope you get the idea. See you in the next installment!

Read More »

glory of the snow

January 4th, 2012 Tags: , , , ,

Dunkle Sternhyazinthe 2008-3-7a

By Hedwig Storch (Own work), CC-BY-SA-3.0, via Wikimedia Commons.

One day in last December was a solstice. That also means, we have another PhantomJS release, this time at its version 1.4, codenamed Glory of the Snow. While it’s hot, check it out from its code repository and look for 1.4 tag and then compile it. Binary packages and executables may be available for download after few days or weeks, so if you don’t want to build from source, please be patient a little bit.

Originally I had a huge plan for this 1.4 release (see issue 226 for details). Unfortunately, since Qt 4.8 had a very long release process, 2 months from the release candidate to the final release, this timing was not in our favor. Still, if you follow the release notes, there are few goodies which I’d like to mention again here.

The first is the new WebServer module. In some use cases, people want to interact with running PhantomJS instance, as opposed to launching it again and again. For the implementation itself, we use the code from Mongoose, an excellent embedded HTTP server library. The addition of this HTTP server module allows such a kind of two-way communication. A shortened example of this module (refer to examples/simpleserver.js for the complete code):

var server, service;
 
server = require('webserver').create();
 
service = server.listen(8080, function (request, response) {
    response.statusCode = 200;
    response.write('<html><body>Hello!</body></html>');
});

Two important notes before you go crazy. First, this is considered experimental. We still don’t have full integration with the rest of Qt and WebKit machinery, check for the issues related to this WebServer module. Second, use it for process communication and not as a production-grade web server.

Another extra goodie in this release is the convenient script to build PhantomJS statically. Some of you just want to use PhantomJS in e.g. a continuous integration server and thus care less about the availability of system-wide Qt libraries (or even when you don’t have the permission to modify the system). The build script, available for Mac OS X and Linux (see the deploy subdirectory) automatically downloads Qt source code, configures Qt suitable for the build, and finally starts the compile. Nothing will clobber your system, everything will be setup locally. On a modern machine, this process (excluding the download) takes about 30 minutes. For the Linux version, if you choose to use Qt 4.8, there are also additional benefits: remote web inspection, X11-less via QPlatformAbstraction (aka Lighthouse), as well as various PDF improvements (borrowed from wkhtmltopdf).

Last but not least, the new features in 1.4 were not possible if KDAB did not assign their highly-skilled engineers to help with the improvements. Kudos to KDAB and we’ll be forever thankful for the contributions!

Now back to work, we have PhantomJS 1.5 to be ready before March 20, 2012.

Read More »

three charms in a year

December 31st, 2011 Tags: , , , ,

Luck comes in threes. And I believe it’s mostly good luck, not bad luck.

This year alone I finally had the time to publish three new open-source projects (one of them is as part of Sencha Labs) at the beginning, middle, and end of the year.

The first in line is PhantomJS, the headless WebKit (based on the Qt port). Since it was announced back in January, it is mentioned in lots of online articles and used in many different projects, from assorted testing frameworks to screen capture service. Full review will be revealed during its first birthday in few weeks. If you like PhantomJS in its first year, expect to love its upcoming second one!

Something I have started back in June was CSS Beautify, a simple JavaScript library to format and indent styles written in CSS. For some cases where you would find it useful, read my brief announcement about this tool.

Last but not least, Esprima. This project is also exciting for me because writing a JavaScript parser in pure JavaScript which hits the compromise of performance and readability has been always in my agenda. For the detailed insight, as well as Esprima’s speed feature, read its project announcement few weeks back.

Another year, another fun!

Read More »

most tweeted

December 25th, 2011 Tags: , , ,

This year alone, Sencha blog corner hosts around 190 blog posts of assorted topics, anything from product announcements, basic learning guides, HTML5 scorecards, to web technologies highlights. The winning blog posts, in terms of how many tweets they got (at the time of this writing), are listed here:

635 Understanding Hardware Acceleration on Mobile Browsers
484 IE10 Preview: HTML5 First Look
416 Motorola Xoom: The HTML5 Developer Scorecard
324 Ext JS 4.0 Final Available Today
279 Previewing Sencha Touch 2: Native Packaging and Performance
204 Sencha Touch 2 Developer Preview
203 Android–Ice Cream Sandwich: The HTML5 Developer Scorecard
200 Introducing PhiloGL: A WebGL Framework from Sencha Labs
188 A Web Developer’s Wishlist for iOS 5
187 iPad 2: The HTML5 Developer Scorecard

Apparently the #1 in the above list, the relation between browser and GPU, is the article I wrote back then, which is essentially a different version of my other explanations on the use of backing store and compositing in WebKit. Thus, there is my shameless plug: even if this year does not seem too magical for me, at least what I wrote did get some interests from some of you.

Now, while this is interesting, the fun part is actually how I came up with the list. While I’m sure it’s entirely possible to use various different API to get the numbers, I was playing with PhantomJS to see if I could do that with a simple site scraping approach. Since the number of tweets, embedded via the Twitter widget, is always displayed in each blog post, that shouldn’t be too difficult, should it? Likely I need to clean up the script before I can include it as one of PhantomJS examples, if at all, so in the mean time just look the complete script at this gist: gist.github.com/1519281.

The gut of the script is very simple. First, load the URL of the blog post and then try to find the iframe containing the embedded Twitter widget. This is accomplished easily with just the following lines of code:

page.open(url, function (status) {
    if (status === 'fail') {
        callback.call(this, count, title);
    } else {
        title = page.evaluate(function () {
            return document.title;
        });
        openWidget(page.evaluate(function () {
            return document.querySelector('iframe.twitter-share-button').src;
        }));
    }
});

Once the corresponding URL of the Twitter widget, completed with all the query string, is obtained, it is now a matter of loading it and extracting the count. This is actually handled in the openWidget() function referred above. The entire function is reproduced here verbatim for your pleasure.

function openWidget(location) {
    var widget = require('webpage').create();
 
    widget.open(location, function (status) {
        if (status !== 'fail') {
            count = widget.evaluate(function () {
                return parseInt(document.querySelector('a#count').textContent, 10);
            });
        }
        callback.call(this, count, title);
    });
}

Pretty straightforward.

Now, for this to work, we also need the list of URLs of every single blog post. Again, while it is likely easy enough to get that list using FeedBurner API (through which Sencha serves the blog feeds), I decided to take a shortcut and use the feed view in my Google Reader. After ensuring that those posts in 2011 are available in the subscribed view, a quick hack with Web Inspector and its Copy as HTML feature gave me the complete list of these blog posts (including short text snippet for each, which we happily ignore). Getting the list of the blog URL is a single-line solution:

grep -Eo '(http://feedproxy\.google\.com/[a-zA-Z0-9~\/\_\-]*)' reader.html

Every item in Google Reader gets its own short URL, in the form of http://feedproxy.google.com/~r/extblog/~3/leusWn2oYnc/, which will redirect to the actual blog post. Fortunately for me, the PhantomJS script illustrated here takes care of that automagically.

Looking forward to seeing more successful 2012 blog posts!

Read More »
« Older Entries
Subscribe to RSS Feed Follow me on Twitter!