Blogging Roller

As FreeRoller users and readers know, FreeRoller is having problems keeping up with the load of 500+ weblogs and the 101-list traffic. Sometimes the main page takes several minutes to load and often the system just collapses under the stress. According to Anthony Eden, who runs FreeRoller, the system is powered by one little eMachines box. I could use that as an excuse, but, regardless of the hardware, the poor performance of FreeRoller reflects poorly on the Roller software.

I considered using a profiler to figure out the root cause of the performance problem, but a profiler is difficult to setup for a Servlet app, there's a learning curve, and there are only so many Roller hours in my weekend. Plus, I had a gut feeling that the Roller main page, index.jsp, was causing the slow-downs.

Looking at the code behind index.jsp, I found that it uses several queries to build a list of weblogs, hit counts, and the last update time for each weblog. The queries are not complex, but because they are executed using an Object-Relational (O/R) persistence framework (Castor) each query results in the creation of one object per row returned. FreeRoller has over 500 weblogs, so each refresh of the index.jsp page results in the creation of thousands of objects. Even with page caching in place to reduce the frequency of refreshes, this has the potential to bog down the whole system.

I rewrote the code behind index.jsp to use only one query and I also added a limit to that query so that a Roller administrator can limit the number of weblogs displayed on the main page. Next, I did some load testing with JMeter. I set up a JMeter test plan to use 3 threads to hit index.jsp, a 20-second ramp up, and a 300 ms delay between requests. I did this testing on a 1.5Ghz Athlon, 768MB RAM box running Windows XP, Java 1.4.1, and Tomcat 4.1.18. Here are the results:

                   Avg.  Dev.  Throughput  Note

1) Old index.jsp 1051ms  8629  120/min     Heavy CPU usage
2) New index.jsp  248ms  1326  248/min     Limit 500 weblogs
3) New index.jsp   93ms   186  247/min     Limit 30 weblogs
4) TC index.jsp    90ms    72  337/min     Tomcat example page

The old index.jsp would peg the CPU at 100% for minutes at a time during cache refreshes. I'm not sure why, but I suspect that multiple requests were simultaneously refreshing the cache (I'm not sure how OSCache hanldles this). During test #1, Roller was so slow as to be almost unusable. Test #2 and #3 show that the new index.jsp is a great improvement. With a limit of 500 weblogs, the CPU usage is very heavy during a cache refresh but doesn't get to 100%. With a limit of 30 weblogs, the cache refresh does not have a noticable effect on the CPU. Test #4 tests the Tomcat examples page, it is included only as a baseline for comparison.

I made the above changes in the Roller 0.9.6 code branch so that they can me applied to FreeRolller right away. I'm still not happy about Roller performance and memory usage. I'm sure there is plenty of room for improvement. My next experiment is implementing the Roller business tier with the Hibernate O/R framework. I'm curious to see how it compares to Castor.