Benchmarking JavaScript Engines - V8, SFX-Nitro, Carakan, Tracemonkey

We were evaluating javascript engines for the purposes of integrating in a prototype product. Chrome-V8, Safari-Squirrel Fish Extreme(aka Nitro) and Firefox Tracemonkey are the three options that we had to consider(just because all three of them are open sourced). There are lot of online resources with comparison study done however all of them are either old or not relevant now. So, thought I would share what we are seeing with respect to speed. There are other metrics that influence the adoption but this article is solely on the speed. I have included Opera results for the completeness. IE is not included for the time constraints that we had and also it being dog slow. 

Setup:

  • Use Dromaeo for the purposes of comparison. It is super easy to run and compare. 
  • Every benchmark is done multiple times and took an average - though variation is very minimal
  • During every run, machine had only 3 apps open. One putty session, one process explorer and the relevant browser. So machine resources are same for all.
  • Run all the benchmarks on Windows box - Dell Poweredge SC 420 - XP, Pentium 4 HT, 2 CPU 2.8GHz, 2GB RAM.
  • Used Macbook Pro 15" to make sure the trend is same. 
  • All the latest release versions: Chrome - 4.1.249.1045 (Build 42898), Safari - 4.0.3(Build 531.9.1), Opera - 10.51(Build 3315), Firefox - 3.6.3
Opera has save-as text file feature. So after all the runs, results are extracted by loading the html page in Opera » save as text file » then run following one-liner to get every run into ":" separated text file so that it is easy to load up in excel. 

more * | egrep ":|runs\/s|txt" | egrep -v "URL:|http:|Origin, Source, Tests:|run on:|::::::|rv:" | perl -ne 'chomp; s/runs/ :runs/g; s/txt/txt: /g; if(/runs/){print "$_ \n";}else{print}' > ../foo.txt

more * is being used above assuming all the text files are saved in the present working directory. 

Sunspider:

Includes well balanced benchmark suite. Covers various areas including math problems, string operations, crypto, raytracers, etc.. 
As seen above, Caracan is the clear winner followed by V8, SFX and TraceM for Sunspider benchmark suite. 

DOM Core:

These tests include setting, getting DOM attributes, DOM traversal, DOM element querying, etc..
SFX is the clear winner followed by V8, TraceMonkey and Caracan for DOM Core suite.

V8 Test:

These include tests like looping, function calls, object manipulation, etc.. 
 
As seen above, V8 is clear winner followed by Caracan, SFX, and Tracemonkey for V8 benchmark suite.

JS Lib:

Fom jQuery, JS Libarary tests include lot of DOM modifications, events, prototypes, jQuery DOM traversal, styling etc...
 
As seen above, V8 is clear winner followed by SFX, Caracan, and Tracemonkey for JS Lib benchmark suite.

I haven't included benchmarks Dromaeo and CSS Selector. Dromaeo; because Caracan results were like 100x on regexp cases which I wanted double check if it is indeed the case. CSS Selector; because each run takes like ~20mins which is quite a bit considering multiple runs.

Conclusion:

If you follow technical documentation on V8, SFX, Caracan and Tracemonkey, you can clearly see similarities in approach each is taking in tackling the problem. Some of the common techniques and modes are creating virtual machines, JIT techniques, inline cache, GC optimizations. Yet there are few differences in each engine. V8 doesn't generate bytecode instead it generates native code directly - similar to compilation for static languages. However it does follow virtual machine techniques under the hood without having two separate steps. SFX generates bytecode and then native code based on the platform it is being run. This approach is similar to LuaJIT where JIT techniques are applied on the bytecode. Caracan seems to take best of both - native code compilation for loops and bytecode interpreter to tackle remaining part of the code. Tracemonkey is slightly different compared to other in that it is the only engine which tries to record/trace as the interpreter kicks in - more details on this here. Jigermonkey is the next version(still in works) to TM which seem to take SFX assembler and combine it to the existing spidermonkey and tracer. 

We have decided to start off with V8 and keep SFX/Nitro as the backup JavaScript engine. Feel free to drop in a comment/question.



No comments:

Post a Comment