JSP Benchmark

The benchmark measures the overhead of several popular servlet and JSP engines. It consists of a simple "Hello, World" servlet and three JSP files: "Hello, world", session, and post. In each case the generated page is identical except for the background color.

The methodology section explains why this benchmark is a valid measurement of servlet engines. The benchmark source is available as benchmark.jar

Data

The following JSP engines were tested:

Resin 1.0 b8 (own web server)
Resin 1.0 b8/Apache 1.3.9
JRun 2.3.3/Apache 1.3.9
ServletExec 2.2b1/Apache 1.3.9
JServ 1.0 final/gnujsp 0.9.8/Apache 1.3.9
JWSDK 1.0 (August release) (own web server)

JSP Engine File Servlet Hello Session Post

Resin 235 231 253 218 194

Resin/Apache 493 172 181 150 142

JRun/Apache 476 51 32 31 30

ServletExec/Apache 498 31 31 30 30

Jserv/Apache 498 41 30 30 29

JWSDK 42 41 12 11 12

Methodology

Skeptical of benchmarks? Good. However, this benchmark is indeed valid for several reasons. First, it primarily measures overhead, the best possible performance for a JSP engine. Second, except for the overhead, JSP applications have identical performance. Third, the benchmark is not cacheable.

Overhead measurements are always valid and often the most useful measurement. It tells you the best you can possibly do, i.e. the speed limit. Because it doesn't exercise any functionality except the overhead, it's impossible to defeat the measurement. RPC mechanisms like CORBA, NFS, or message passing kernel OS often quote the 'null-RPC' measurement.

The bulk of a servlet's work is identical no matter which JSP engine is chosen. The Java code runs at the same speed and the database drivers are identical. So creating a "more realistic" benchmark would only add a constant time to each request.

The second point does become important when comparing JSP/Java to JSP/JavaScript or Perl or PHP or ASP. This benchmark would be less effective in comparing technologies with different languages and drivers. It still would give a sureful overhead result, but would not give a good idea of how real applications compare. In addition, because the JDK is the same, the JSP engines can't do the equivalent of optimizing out loops.

Third, the benchmark isn't cacheable without heroic measures. When comparing computers or static web servers, the performance when the results come from a cache are better and unrepresentitive of real results. That's one reason the sieve and n-queens benchmark are bad, and why SpecWeb goes to great lengths to defeat web server caching. (Of course, in the last case, many sites can fit the most popular pages into a memory cache, so the memory results may actually be more indicative of the real performance.)

Finally, tweaking configurations typically only gets at most 20% improvement. Resin is three times faster. No amount of tweaking will change that significantly. For example, JDK 1.1.7 was used because it's the most stable on the Linux platform and because all the engines ran successfully with it. Internally, I did spot checks with JDK 1.2 and the numbers were similar. The full results for JDK 1.2 will be available shortly; time pressures limited their inclusion.

Conclusion

This benchmark shows that JSP can approach static page performance. The comparison between different JSP engines is really less important, although Resin's performance is certainly personally gratifying. A perception still exists that Java is a great programming environment but hobbled by performance. The perception is no longer valid.

The near-static performance means that many sites, who never considered JSP or servlets because of performance, can now take advantage of the Java platform's reliability and ease of programming benefits.

Benchmarks

XSLT Benchmark