I've been playing recently with Jython and decided to do some quick and dirty benchmarking with pystone. In order to have a reference, I first tested cPython 2.6, with an increasing numbers of loops (I thought this may be relevant as Jython should start to profit from the JIT only after some time).
(richard garibaldi):/usr/local/src/pybench% python ~/tmp/pystone.py
Pystone(1.1) time for 50000 passes = 1.04
This machine benchmarks at 48076.9 pystones/second
(richard garibaldi):/usr/local/src/pybench% python ~/tmp/pystone.py 500000
Pystone(1.1) time for 500000 passes = 10.33
This machine benchmarks at 48402.7 pystones/second
(richard garibaldi):/usr/local/src/pybench% python ~/tmp/pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 19.6
This machine benchmarks at 51020.4 pystones/second
As you can see, cPython behaves consistently: the time it takes to complete the test increases linearly to the number of loops. Knowing this, I started testing Jython.
(richard garibaldi):/usr/local/src/pybench% jython ~/tmp/pystone.py
Pystone(1.1) time for 50000 passes = 2.29807
This machine benchmarks at 21757.4 pystones/second
(richard garibaldi):/usr/local/src/pybench% jython ~/tmp/pystone.py 500000
Pystone(1.1) time for 500000 passes = 10.931
This machine benchmarks at 45741.4 pystones/second
(richard garibaldi):/usr/local/src/pybench% jython ~/tmp/pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 107.183
This machine benchmarks at 9329.86 pystones/second
During the first run Jython runs rather lousily in comparison to its C brother. When increased the number of loops it started feeling better, coming close to cPython, like my initial hypothesis predicted. Note that the number of loops increased 10 times, but it took Jython only about 5 times longer to complete them. So, as you imagine, I was expecting that Jython would really rock in the final test. To my great disappointment, however, it did really bad: more than twice slower than in the initial run.
What are your hypotheses: why does Jython behave such an inconsistent manner? Could it be that GC is kicking in at some moment, and taking a lot of time? I've looked at PyStone's code and garbage collection doesn't seem to be turned off, but I would expect Java's GC to be at least as good as Python's... Do you think this slowing down is permanent, or it will go away at some point after increasing the number of loops? How shall Jython behave in a really long running processes?
EDIT: unfortunately, I get java.lang.OutOfMemoryError if I increase the number of loops to 2 million...
(Of course, Jython is still beta, so it should get better in the final release.)
I am using Jython 2.5b1 (trunk:5903:5905, Jan 9 2009, 16:01:29), Java(TM) SE Runtime Environment (build 1.6.0_07-b06-153) and Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_07-b06-57, mixed mode) on MacOS X 10.5.
Thanks for your answers.
Benchmarking a runtime environment as complex as the JVM is hard. Even excluding the JIT and GC, you've got a big heap, memory layout and cache variation between runs.
One thing that helps with Jython is simply running the benchmark more than once in a single VM session: once to warm up the JIT and one or more times you measure individually. I've done a lot of Jython benchmarking, and unfortunately it often takes 10-50 attempts to achieve a reasonable time
You can use some JVM flags to observe GC and JIT behavior to get some idea how long the warmup period should be, though obviously you shouldn't benchmark with the debugging flags turned on. For example:
% ./jython -J-XX:+PrintCompilation -J-verbose:gc
1 java.lang.String::hashCode (60 bytes)
2 java.lang.String::charAt (33 bytes)
3 java.lang.String::lastIndexOf (156 bytes)
4 java.lang.String::indexOf (151 bytes)
[GC 1984K->286K(7616K), 0.0031513 secs]
If you do all this, and use the HotSpot Server VM, you'll find Jython slightly faster than CPython on pystone, but this is in no way representative of Jython performance in general. The Jython developers are paying much more attention to correctness than performance for the 2.5 release; over the next year or so with a 2.6/2.7/3.0 release performance will be more emphasized. You can see a few of the pain points by looking at some microbenchmarks (originally derived from PyPy) I run.