Blogging Roller

I've been working on improving the performance of the Roller persistence tier. I've been doing this work in the old Roller 0.9.7 branch to make it easy for FreeRoller to deploy the improvements. Here are some details on the improvements.

Roller uses an O/R mapping framework persist objects in a relational database. Before the improvements, every call into the Roller persistence layer would open, use, and close a persistence session (either a Hibernate Session or a Castor Database). Roller often makes multiple persistence layer calls to process a web request and, so, Roller would often open, use, and close multiple persistence sessions per web request. A number of people pointed out that this is a bad thing.

To fix this problem, I changed Roller to ensure that each incoming web request gets it's very own persistence session and all persistence layer calls made on behalf of that request use that same persistence session. That means that there is, at most, one persistence session per incoming request. Persistence sessions don't hang around, each one lasts for the duration of one web request only.

These change improved Roller's performance, but not by much. Here are the JMeter test results. As always, comments, suggestions, and criticisms welcome. I think it is interesting that Resin performed better than Tomcat on the baseline test, but not the Roller test (a sign that I screwed something up?).

Update: FreeRoller is deploying these changes now. Even though the performance gains are minimal, I'm hoping the load and the number of MySQL threads will be reduced. I'll be playing with Hibernate caching and the new OSCache beta release tonight to see if they can offer any additional improvements.

It's a nice coincidence that the very week that I have to dig into the Struts Validator, two articles appear by Erik Hatcher on XDoclet and Struts. After reading these articles and browsing Matt Raible's AppFuse project code, I was able to fit the pieces together and get XDoclet to generate the right struts-config.xml and validation.xml entries for my form beans. Below are links to Erik's articles and some other resources I found useful in investigating Struts Validation:

• Erik Hatcher: Form Validation and Action Mapping
• Erik Hatcher: Exploit XDoclet's Struts Capabilities
• Matt Raible: AppFuse project
• <a href= "http://www.raibledesigns.com/page/rd/20030226#struts_validator_validating_two_fields"> Matt Raible: How To Add a TwoFields Validator
• Struts website: Struts Validator Guide
• Chuck Cavaness's Using the Validator Framework with Struts

Seems like everytime I try to use XDoclet, I have to tweak one of the XDoclet template files to work around a bug. This time was no exception. I found that XDoclet would only generate the right entries in struts-config.xml if my form beans extend ActionForm, but to use the Struts Validator, I need my form beans to extend ValidatorForm. The only way I found to get around this problem was to copy struts_config_xml.xdt out of the XDoclet xdoclet-apache-module-1.2b4.jar file and to change the first forAllClasses like so (my change in bold):

  <!-- ========== Form Bean Definitions ============== -->
  <form-beans>

   <XDtClass:forAllClasses type="org.apache.struts.validator.ValidatorForm">
   <XDtClass:forAllClassTags tagName="struts:form" superclasses="false">
    <form-bean
      name="<XDtClass:classTagValue tagName="struts:form" paramName="name"/>"
      type="<XDtClass:fullClassName/>" />
   </XDtClass:forAllClassTags>
  </XDtClass:forAllClasses>

I believe there needs to be one of those forAllClassses loops for each type of form: ActionForm, ValidatorForm, and ValidatorActionForm. This does not appear in the current XDoclet CVS for <a href= "http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/xdoclet/xdoclet/modules/apache/src/xdoclet/modules/apache/struts/resources/struts_config_xml.xdt?rev=1.11&content-type=text/vnd.viewcvs-markup"> struts_config_xml.xdt. My chance to get a patch in?

Congratulations and good luck to fellow RTPBlogger Joe Gregorio who recently launched his new company BitWorking, Inc. to do custom systems development.

Big round of applause for Anthony Eden and Min Idzelis who have just joined the Roller project. Hip, hip, hoooooray! By the way, Min lives right in the Triangle. I'll have to convince him to come out to the next RTP Bloggers lunch.

Charles Miller - Version Numbers and You: Too many Open Source projects treat Version 1.0 as some kind of Holy Grail that can only be reached when the project is perfect. I find that highly annoying, because it makes it really, really difficult to tell a sketchy alpha from production code that is just still in pre-1.0 because the author wants it to do everything.

If we add multiple categories per post, I'd be happy calling the next release Roller 1.0 rather than Roller 0.9.8.

Roller has been honored by the BileBlog. Most of Hani's criticisms are valid, but none new or especially insightful, at least not to me. I'm a little surprised he did not attack the sucky UI as it is a prime target. Also, it is interesting that he zoomed in on the multiple connections per request problem, I think he had an inside tip ;-) That is what we are working on this week. I sincerely hope he is correct and that fixing the multiple connection per request (enhancement #9 from my earlier post) solves the FreeRoller current performance problems.

As you may or may not remember, I started developing Roller, in part, to teach myself about database driven web applications in the world of Java and J2EE. So, the history of Roller performance is also the history of my school of hard knocks education in Java webapp performance. I'm not sure if anybody can learn anything of value from this, but I feel like reviewing the story, and this is my blog, so that's what I'm gonna do. If you don't like it, you know where the back button is. Here is the story so far.

In the beginning, I implemented Roller using EJB. Every Roller business object was an Entity Bean and every Roller business tier interface was a Session Bean. I learned a lot about EJBs and XDoclet and everything was just fine, but Roller, or Homeport as I used to call it, took 20 seconds to render a weblog page. I studied the various EJB patterns and realized that returning a collection of Entity Beans from the business tier was not a good idea. I learned about the Data Access Object and Value Object patterns. I discovered that XDoclet could generate Value Objects (which Sun now calls Data Transfer Objects) for me, and I changed the business tier interfaces to return the light-weight data objects rather than Entity Beans. Much better! That was performance enhancement #1.

Once I had the business tier interfaces returning data objects, I found that it was easy to convert Roller over to use Castor for persistence. I was tired of waiting for EJB containers to start, tired of waiting for EJBs to deploy, and generally disillusioned with EJB. So, I created Castor mappings to map the XDoclet generated Value Objects to database tables and switched over to Castor. Again, much better. That was performance enhancement #2.

When I tried to get Roller to display my bookmarks page, which has a hundred or more bookmarks, I still found that Roller was way too slow. Rendering a page was taking 10 seconds or so. I realized that I was not taking advantage of database connection pooling. Configuring database connection pooling was performance enhancement #3.

I'm not sure about the next chapter, but I believe that it happened this way. I was forced to switch from Resin to Tomcat and Tomcat was slow (back then it was slow, now it screams ;-). I found that OSCache could allow me to hide, almost completely, this problem. OSCache was enhancement #4.

I'm a little fuzzy about this one too, but here is what I remember. One day I read that some guy named Anthony Eden had decided to set up Roller and to offer free accounts on a site called FreeRoller. I figured he would close FreeRoller off after reaching 10 or 20 users. He didn't. He kept going and going. Now I'm not sure if he is ever going to stop. Anyhow, Roller started running out of memory, so we adjusted to session timeout to a lower value and I moved the Roller business tier implementation from being a session thing to being an application thing. Those changes, call them enhancement #5, fixed the out-of-memory problems.

FreeRoller did pretty well for a while, especially considering the el-cheapo little eMachines box that Anthony started with. Eventually, it got bogged down again. I was starting to think that our persistence framework was at fault and I had heard that Hibernate was much faster than Castor, so I switched to Hibernate. Hibernate seemed to be about twice as fast. I'll call that enhancement #6.

Around this time, JavaLobby offered to host FreeRoller. FreeRoller moved from Anthony's little eMachines box to a big honkin' JavaLobby server. Everybody danced a wild fandango of glee and revelled in the new found speed and snappiness of Roller. FreeRoller != Roller, but for the sake of this story, we'll call that enhancement #7. The fast machine worked great for a while, but eventually it too started to slow down. At first, users did not notice the problem, but Rick Ross and Matthew Schmitt complained that Roller's load was way too high. After some investigation, we figured out that the RSS Servlet was doing more work than it should. So, enhancement #8 was to make sure that RSS feeds are cached and that the IF-MODIFIED-SINCE header was being handled and handled efficiently. That fixe brought the Roller load down to reasonable levels.

That brings us to today. The number of FreeRoller users and readers has grown and FreeRoller is again creaking and straining under the pressure. I hadn't noticed this because I was so busy with other things, but Anthony noticed, and so did Lance. Lance did some research and theorized that the problem is that Roller uses too many database connection to process each request. Each Roller weblog page is rendered by a user-edited Velocity script with a dozen or more embedded macros. Each of these macros makes one or more calls to the Roller business tier and each Roller business tier method opens a connection, performs some database operation, and closes the connection. Roller uses database connection pooling, but even so, grabbing and releasing a connection dozens of time for each request can't be a good thing.

Long story long, Lance modified Roller to use one connection per request. Now we are testing Roller and if all goes well, I'll backport the changes to the Roller 0.9.7 branch so that they can easily be applied to FreeRoller this weekend. I'm keeping my fingers crossed and hoping that enhancement #9 will hold us over for a couple of months while we figure out enhancement #10. I've learned a lot from Roller and I suspect that I have a lot more to learn.

Java World: JSP [document use] XML-compliant include and forward actions, as well as custom tags, so making the entire document XML-compliant leads to increased code consistency.

Jason Carreira: Scaling doesn't have anything to do with whether your objects are remote. How often do you really have one web request cross JVM boundaries? The whole Servlet-tier and Business Object tier topography has been pretty well discredited. It doesn't scale to have to make remote calls.

I've been investigating web application testing tools, both for unit testing of web components such as Struts Actions and for functional testing. I read about JUnit extensions Cactus and HttpUnit in Hightower and Lesieki's book Java Tools for Extreme Programming. I learned about StrutsTestCase and JUnitDoclet from Matt Raible's gold-mine of example Ant code: the AppFuse project. That was enough to get me started.

I put together a simple test app with Cactus and StrutsTestCase and was pleased to find how easy it is to invoke Cactus from an Ant script and have Cactus start Jetty, run my tests within the Jetty, and shutdown Jetty when the tests complete. Cactus is going to make it really easy for us to unit test our Servlets, Filters, JSP pages, and Struts Actions. Cactus uses HttpUnit and HttpUnit makes it easy to make sense out of the responses that come back from the web layer, see the HttpUnit cookbook page for an example if this. I wish I had done this a long time ago, how did I live without this stuff!

So, I'm all set for unit testing. Now I need to figure out what to do for functional testing. HttpUnit, MaxQ, and Canoo look promising. I found the bewilderingly long list of functional testing tools at opensourcetesting.org. What open source or commercial tools have you used for functional testing of web applications?

I finally submitted the last of my chapter edits this week for the Apress JSP book. Now I have my evenings back and can get back to the kids, to clearing a giant Netflix queue, cleaning out the garage, and to working on Roller.

I've been too busy do anything other than watch the emails fly by, but Min has been busy. He added full weblog search to Roller via Lucene. Take a look at Matt's write-up for details.

Good stuff from Mr. Lowem: <a href= "http://freeroller.net/page/lowem/20030718#more_roller_annoyances">Minor Roller anoyances and <a href= "http://freeroller.net/page/lowem/20030718#more_roller_annoyances">More Roller Annoyances. Interested in improving Roller? Join us on the Roller development mailing list. There are lots of little bugs to fix and minor improvements to be made, great for those getting started with server-side Java, but there are probably some deeper architectural issues for those interested in performance and scalability.

Slashdot picked up Andy's <a href= "http://linuxintegrators.com/hl30/blog/general/?permalink=LGPL+clarification.html">LGPL is viral for Java story and provoked a response from the Free Software Foundation (FSF):

<a href= "http://developers.slashdot.org/developers/03/07/17/2257224.shtml"> FSF Executive Director, Brad Kuhn LGPL's S. 6 allows you to make new works that link with the LGPL'ed code, and license them any way you see fit. Only the LGPL'ed code itself must remain Free. Such 'client code' can even be proprietary; it need not be LGPL'ed.

I'm so confused. Is that the final word? He didn't mention Java specifically. He is not a lawyer. I don't think this is going to be the final word for Apache. I'm starting to think that FSF wants this to be a confusing issue.

Matt's taking the day off to celebrate his birthday. Have a good one Matt.

<img src="http://www.rollerweblogger.org/resources/roller/D071503b0017ww.jpg" alt="Linus runs down the boardwalk towards the beach and a landing ultra-light aircraft" />

(C) 2003 Charles S. Johnson

Andy Oliver: I clarified the LGPL in Java issue with the FSF and posted to poi-dev. According to the FSF, the LGPL is indeed viral towards Java code. Thus, in essence GPL and LGPL are the same for Java. You could still find that the copyright holder of a particular software package held a different view and is willing to state that explicitly, but it looks like the Apache Software Foundation and Sam were right and I was wrong.

That really sucks. Not the part about Andy being wrong, that happens all the time. I'm talking about the now confirmed viral nature of the LGPL when applied to Java. I believe that Roller uses a couple of LGPL components, JSPWiki for example. I guess this means that either those components should be removed or Roller should switch to a dual GPL and APL license.

This year we are vacationing in Emerald Isle, NC, about a three hour drive from home. My parents rented a big house about a block from the ocean and invited us to come along. With three kids, having both parents and grand-parents around is really, really convenient. There is always somebody available to take the big kids to the beach, somebody to stay home when the littlest guy needs to nap, and somebody to babysit so Andi and I can drive over to Beaufort for dinner one night. Dad, the expert photographer and physical chemist extraordinaire, brought along his big honking digital SLR camera and his assortment of telephoto lenses, so I don't even have to worry about taking pictures.

This is all great and the only thing really causing any stress is the ProJSP book, but I don't think I'll have to miss a day at the beach to wrap up my changes to my two chapters. My Dad and I have our laptops setup in the kitchen and the only thing that really interferes with my works is the constant derision and calls of "hey nerd-boys" and "get a life" from our wives. This really doesn't bother me at all. I always arrange to bring a laptop along on vacation. I'm not like Raible. My idea of a vacation is tinkering with all things digital, which, coincidentally, is also my idea of work. I'm the last of the great indoorsmen.

I do go outside occaisionally. I'm writing this sitting in a rocking chair on the porch overlooking the ocean. We watched a little motorized para-glider fly by just before sunset. Now we are watching some impressive amateur fireworks going up accross the street and trying to keep the boys from stepping on the tiny tree frogs that appear after dark. Damn, I can't see the keyboard anymore. Time to go back inside.

Via Corante: Mena Trott on AOL's blog offering: "This isn't just some message board with a blogging label slapped on -- the AOL Journals team is taking the time and effort to get this right and that's highly commendable."

MozillaZine: It has been learned through public and private sources that AOL has cut or will cut the remaining team working on Mozilla in a mass firing and are dismantling what was left of Netscape (they've even pulled the logos off the buildings). Some will remain working on Mozilla during the transition, and will move to other jobs within AOL.

I'm not being entirely fair. AOL gave Mozilla a nice little send off and a nice wad of cash to move Mozilla forward.

I was unable to use the early Tomcat 5.0 alphas when I was working on Professional JSP 2.0 and I've heard plenty of complaints about Tomcat 5's extreme instability, but the latest alpha seems to work just fine. I downloaded it, loaded my Data Access for Web Applications example in, and it just worked! I was also happy to find that the Sysdeo Tomcat plugin has been updated to support the Tomcat 5.X line.