| Benchmark | Result | Scale | What it measures |
|---|
asm.js appears in one test in Octane, and in several tests in JetStream, which is great, but several aspects of asm.js performance are not fully measured in those. In particular, asm.js often appears as large (you might even say massive) source files, which can have different performance characteristics than the typical small programs appearing in most benchmarks. For example, a very large codebase may contain very large functions, which due to their size are difficult to optimize efficiently (either not fully optimized, or optimized slowly in a noticeable way), or just the sheer number of functions may be very high and cause the browser to pause as the codebase is parsed and starts to execute. Such very large codebases can therefore bring new challenges to JavaScript engines (or rather, more extreme versions of familiar challenges), and it is important to measure performance on them because they are showing up with increasing frequency on the web (for example, as native plugins are fading out, game engine companies like Unity and Epic are starting to compile their large codebases to asm.js). For these reasons, the Massive benchmark includes several very large codebases (Poppler, Bullet, SQLite, etc.), and measures throughput as well as responsiveness, variability and startup time (see details below).
The Emscripten benchmark suite evolved over time in order to benchmark Emscripten itself, and therefore mainly focused on throughput, and is runnable in both shell and browser. Massive, on the other hand, tests not just throughput but also browser responsiveness and other factors that only make sense when running in a browser, things not measured by the Emscripten benchmark suite (or by the main JavaScript benchmarks).
Main Thread Responsiveness measures the user experience as a large codebase is loaded. What is tested is whether the main thread stalls as the codebase is prepared and executed for a short while. The score here can be improved by parsing the code off the main thread, for example. This does not measure how much time is spent, but only how responsive or unresponsive the user experience is (how much time is spent is measured by Preparation, and to some extent Throughput). Technically, we measure responsiveness by seeing if events on the main thread execute at the proper interval (as when the main thread stalls, it stalls both the user experience and other events).
Throughput measures how fast a large computational workload runs. This is what is typically measured by benchmarks. Massive's throughput tests focus on very large real-world codebases.
Preparation measures how long (in wall time) is spent to get a codebase ready to execute, before executing any of it. This measures how much time passes between adding a script tag with that code and being able to call the code (this may or may not cause a user-noticeable pause, depending on whether it is parsed on or off the main thread; Main Thread Responsiveness tests that aspect). "Preparation" is basically all the time before code is actually able to run; that may include parsing, conversion to bitcode, JIT compilation, etc., depending on the JS engine.
Variance measures how variable the frame rate is in an application that needs to run in each frame (this is important in things like games, which much finish all they need every 1/60 of a second, in order to be smooth). Specifically, we run many frames and then calculate the statistical variance and worst case. Note that one VM might have a much faster overall frame rate than another, but also more variance: in general, given two VMs with the same average, the one with less variance is "better" since it's smoother. But given a different mean, things are less clear (perhaps we are happy to get some average slowdown in order to reduce variance which can cause noticable but rare pauses?). Hence we measure variance separately from throughput (which is a measurement of the total speed, and is proportional to the average).
Most of the tests, in particular the throughput ones, are generally very consistent, as we run a deterministic workload in a web worker, which minimizes outside noise. We also run a few repetitions and average the results. However, in particular the Main Thread Responsiveness tests need to run on the main thread, and they involve DOM events like adding a script tag, setInterval, etc., which can be fairly variable. We run a larger amount of repetitions on those tests to average out the noise, but even so they appear to be less consistent between runs on some browsers.
When we see the results of a test are too variable, we mark it with "(±X%!)" next to the score. The cause of such variability might be something else on your machine (perhaps a background indexing service happend to use a CPU core during a test, etc.), or it might be that the browser behaves unpredictably for some reason.