Using Lithium to reduce bugs in Firefox

Lithium has been used it to make reduced testcases for hundreds of crashes and assertion bugs in Firefox. Here's how.

Preparing the testcase

If the testcase is pure JavaScript with no DOM interaction, use the command-line JavaScript Shell instead of Firefox.

If the testcase has to run in Firefox:

Make the testcase cause Firefox to exit when finished, using goQuitApplication() in examples/mozilla/quit.js.
Tell Lithium what section of the testcase file it is allowed to reduce, by adding comment lines containing "DDBEGIN" and "DDEND" to the file. This prevents Lithium from needlessly introducing syntax errors or removing the goQuitApplication() call.

Running Lithium

Lithium's general syntax is:

    ./lithium.py (interestingness-test) (file-to-reduce) [extra arguments for the test]

You'll usually be able to use one of the tests that comes with Lithium:

Test	Parameters	What it tests
outputs.sh	app, X	(App run with the testcase) outputs X
outputs1.sh	app, X, Y	(App run with the testcase) outputs X but not Y
crashes.py	app, [timeout=300]	(App run with the testcase) crashes
hangs.py	app, [timeout=30]	(App run with the testcase) does not exit within the timeout, in seconds

For example, suppose you have a large file called boom.html that triggers an assertion in debug builds of Firefox. To make a reduced testcase, you might use something like:

    ./lithium.py ./outputs.sh boom.html fxdebug/firefox-bin "ASSERTION: index out of range"

Lithium will try to remove as many lines from boom.html as possible while still causing Firefox to print that assertion message.

Tips

All operating systems

Before testing a given build of Firefox, make sure that build of Firefox is the one you ran most recently. Otherwise, Firefox will restart in a way that makes it seem to Lithium like Firefox has exited (see bug 271613). This kind of restart can also cause a testcase filename to be treated as a hostname (bug 396003).
Before testing crashes, turn off Firefox's Breakpad crash reporter.
Before testing crashes, turn off session restore.
When testing crashes, it's usually better to use outputs1.sh than crashes.py. Lithium often gets confused by random startup crashes if you use crashes.py; it thinks an intermediate testcase crashes Firefox when the crash was just a random startup crash.
You might want to edit the test files to hard-code your Firefox path and/or -P profilename.

Mac

Before testing crashes in GUI apps, turn off the OS crash dialog by running CrashReporterPrefs and set it on "Server" mode. Remember to set it back to "Developer" mode when you're done.
When using tests based on timed_run.py (crashes.py / hangs.py), give it "firefox-bin", not "firefox". The "firefox" shell script runs firefox-bin in a way that prevents timed_run.py's hang protection from killing firefox-bin.
Since you'll be using your computer for other things while Lithium reduces the testcase, you'll probably want to work around bug 320746 (pressing the Alt key while Firefox is starting triggers the Safe Mode dialog, even if Firefox doesn't have focus) by commenting out that code in your Firefox tree.
If Firefox opens two windows, you're probably hitting bug 366009. Apply the patch there.

Windows

To turn off the crash dialog, use regedit to change the following keys in [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug]:
```
     "Auto"="1"
     "UserDebuggerHotKey"=dword:00000000
     "Debugger"=""
```
Be sure to keep the previous value for Debugger around so you can set it back.