We wanted to test runtime performance on an app that performed some operation that most C++ developers would readily understand. In addition, we wanted an app that was:
For the runtime tests, we used GNU Bison 1.25. Bison takes a grammar file as a command line parameter and generates C code that implements a parser for that grammar. Bison contains 11,000 lines of C code; compiled with Microsoft Visual C++ debugging information, bison.exe is 163 KB.
The goal was to compare runtime performance for typical debugging sessions. Execution time was measured by printing the time of day at the top and bottom of main(). Execution times were compared for a debug Visual C++ build, HeapAgent 3.0, Purify 4.0, and BoundsChecker Pro 4.0 with and without compile-time instrumentation (CTI).
Bison is available at:
http://www.cdrom.com/pub/gnu/bison-1.25.tar.gz
For this benchmark, we chose the C++ grammar file that is distributed with GNU GCC. This file, extract.y, can be found at:
http://www.cdrom.com/pub/gnu/gcc-2.7.2.tar.gz
Note Bison uses long file names, so you can only run this test on Windows NT or Windows 95.
Also note You can extract the contents of these files using WinZip, which is available at:
http://www.winzip.com
If you'd like to run these tests yourself, you can download an additional file from our Web site that contains a makefile, an INI file for the BoundsChecker compile-time instrumentation test, and a readme file that contains the description you're now reading:
http://www.microquill.com/prod_ha/source/ha_bench.zip
We ran the tests in the following test environment:
We modified main.c to print the time of day at the top and bottom of main():
main.c
======
24d23
< #include <time.h>
48d46
< { time_t curtime;time(&curtime);puts(ctime(&curtime)); }
90d87
< { time_t curtime;time(&curtime);puts(ctime(&curtime)); }
The other change to the standard distribution was the addition of makefile, which is included in ha_bench.zip, as noted above. The command line to build the executable is:
nmake CC=cl
Run Bison with only the Visual C++-supplied memory debugging:
bison extract.y
Start HeapAgent and open bison.exe. Set the command line arguments to extract.y. Use all default settings.
Start Purify and Run bison.exe extract.y. Use all default settings.
Start BoundsChecker and open bison.exe. Set the arguments to extract.y. Set the Error Detection mode to Quick in BoundsChecker Settings.
For the BoundsChecker compile-time instrumentation test, you need to recompile bison.exe. We created an INI file, bc_cti.ini, for BoundsChecker to use during this recompile (bc_cti.ini is included in ha_bench.zip, as noted above):
checking_level full checking_uninit_compile on compiler_fault_recovery on precompiled_header off
Then rebuild:
nmake clean nmake CC="bcompile -Zop bc_cti.ini"
Finally, run bison.exe under BoundsChecker as described in the previous test.
Each test was performed three times. The displayed starting time was subtracted from the displayed ending time to find the execution time. The three execution times were then averaged to arrive at the following results:
Test mm:ss Factor ================================================================== Visual C++ 00:04 1x HeapAgent 00:05 1.25x Purify 00:44 11x BoundsChecker Pro (without CTI--"Quick" mode) 06:56 104x BoundsChecker Pro (without CTI--"Normal" mode) 06:57 104x BoundsChecker Pro (without CTI--"Maximum" mode) 57:43 865x BoundsChecker Pro (with CTI--"Quick" mode) 10:35 158x BoundsChecker Pro (with CTI--"Normal" mode) 15:45 236x BoundsChecker Pro (with CTI--"Maximum" mode) 66:18 994x
We recompiled some of the MFC source code (approximately 110K lines in 190 files) and measured the amount of time required by Purify and by BoundsChecker Pro with CTI. We performed three runs for each; the results listed below are the average times:
Product Total compile/link time Factor
========================================================
Visual C++
debug build 23 minutes, 25 seconds 1x
HeapAgent 23 minutes, 25 seconds 1x
(works on debug build, so
no recompile is required)
Purify
object-code
insertion 24 minutes, 20 seconds 1.04x
BoundsChecker Pro
compile-time
instrumentation 100 minutes, 55 seconds 4.31x
Note To enable stack checking, HeapAgent requires the /Ge compile flag, which activates stack probes for every function call that requires storage for local variables.
If you'd like to try the BoundsChecker Pro recompile test yourself, or if you'd like to see which compile flags we used, you can download the batch file we used to compute these timings. (No special files are required for HeapAgent or Purify.)
http://www.microquill.com/prod_ha/source/cti_test.zip