Dave Johnson on open web technologies, social software and software development
« More JSP best practi... | Main | Relegated to the... »
As you may or may not remember, I started developing Roller, in part, to teach myself about database driven web applications in the world of Java and J2EE. So, the history of Roller performance is also the history of my school of hard knocks education in Java webapp performance. I'm not sure if anybody can learn anything of value from this, but I feel like reviewing the story, and this is my blog, so that's what I'm gonna do. If you don't like it, you know where the back button is. Here is the story so far.
In the beginning, I implemented Roller using EJB. Every Roller business object was an Entity Bean and every Roller business tier interface was a Session Bean. I learned a lot about EJBs and XDoclet and everything was just fine, but Roller, or Homeport as I used to call it, took 20 seconds to render a weblog page. I studied the various EJB patterns and realized that returning a collection of Entity Beans from the business tier was not a good idea. I learned about the Data Access Object and Value Object patterns. I discovered that XDoclet could generate Value Objects (which Sun now calls Data Transfer Objects) for me, and I changed the business tier interfaces to return the light-weight data objects rather than Entity Beans. Much better! That was performance enhancement #1.
Once I had the business tier interfaces returning data objects, I found that it was easy to convert Roller over to use Castor for persistence. I was tired of waiting for EJB containers to start, tired of waiting for EJBs to deploy, and generally disillusioned with EJB. So, I created Castor mappings to map the XDoclet generated Value Objects to database tables and switched over to Castor. Again, much better. That was performance enhancement #2.
When I tried to get Roller to display my bookmarks page, which has a hundred or more bookmarks, I still found that Roller was way too slow. Rendering a page was taking 10 seconds or so. I realized that I was not taking advantage of database connection pooling. Configuring database connection pooling was performance enhancement #3.
I'm not sure about the next chapter, but I believe that it happened this way. I was forced to switch from Resin to Tomcat and Tomcat was slow (back then it was slow, now it screams ;-). I found that OSCache could allow me to hide, almost completely, this problem. OSCache was enhancement #4.
I'm a little fuzzy about this one too, but here is what I remember. One day I read that some guy named Anthony Eden had decided to set up Roller and to offer free accounts on a site called FreeRoller. I figured he would close FreeRoller off after reaching 10 or 20 users. He didn't. He kept going and going. Now I'm not sure if he is ever going to stop. Anyhow, Roller started running out of memory, so we adjusted to session timeout to a lower value and I moved the Roller business tier implementation from being a session thing to being an application thing. Those changes, call them enhancement #5, fixed the out-of-memory problems.
FreeRoller did pretty well for a while, especially considering the el-cheapo little eMachines box that Anthony started with. Eventually, it got bogged down again. I was starting to think that our persistence framework was at fault and I had heard that Hibernate was much faster than Castor, so I switched to Hibernate. Hibernate seemed to be about twice as fast. I'll call that enhancement #6.
Around this time, JavaLobby offered to host FreeRoller. FreeRoller moved from Anthony's little eMachines box to a big honkin' JavaLobby server. Everybody danced a wild fandango of glee and revelled in the new found speed and snappiness of Roller. FreeRoller != Roller, but for the sake of this story, we'll call that enhancement #7. The fast machine worked great for a while, but eventually it too started to slow down. At first, users did not notice the problem, but Rick Ross and Matthew Schmitt complained that Roller's load was way too high. After some investigation, we figured out that the RSS Servlet was doing more work than it should. So, enhancement #8 was to make sure that RSS feeds are cached and that the IF-MODIFIED-SINCE header was being handled and handled efficiently. That fixe brought the Roller load down to reasonable levels.
That brings us to today. The number of FreeRoller users and readers has grown and FreeRoller is again creaking and straining under the pressure. I hadn't noticed this because I was so busy with other things, but Anthony noticed, and so did Lance. Lance did some research and theorized that the problem is that Roller uses too many database connection to process each request. Each Roller weblog page is rendered by a user-edited Velocity script with a dozen or more embedded macros. Each of these macros makes one or more calls to the Roller business tier and each Roller business tier method opens a connection, performs some database operation, and closes the connection. Roller uses database connection pooling, but even so, grabbing and releasing a connection dozens of time for each request can't be a good thing.
Long story long, Lance modified Roller to use one connection per request. Now we are testing Roller and if all goes well, I'll backport the changes to the Roller 0.9.7 branch so that they can easily be applied to FreeRoller this weekend. I'm keeping my fingers crossed and hoping that enhancement #9 will hold us over for a couple of months while we figure out enhancement #10. I've learned a lot from Roller and I suspect that I have a lot more to learn.
Dave Johnson in Roller
07:33PM Jul 28, 2003
Comments [17]
Tags:
Roller
This is just one entry in the weblog Blogging Roller. You may want to visit the main page of the weblog
Below are the most recent entries in the category Roller, some may be related to this entry.
Posted by Lance on July 29, 2003 at 12:02 PM EDT #
Posted by Matt P on July 29, 2003 at 12:44 PM EDT #
Posted by Gerhard on July 29, 2003 at 01:16 PM EDT #
Posted by Gerry on July 29, 2003 at 02:20 PM EDT #
Posted by Dave Johnson on July 29, 2003 at 02:36 PM EDT #
Posted by Anthony Eden on July 29, 2003 at 02:42 PM EDT #
Posted by Anthony Eden on July 29, 2003 at 02:43 PM EDT #
Posted by Lance on July 29, 2003 at 03:36 PM EDT #
Posted by Nadeem Bitar on July 29, 2003 at 06:28 PM EDT #
Posted by Jason Carreira on July 29, 2003 at 07:24 PM EDT #
Posted by Nadeem Bitar on July 29, 2003 at 07:51 PM EDT #
Posted by Dave Johnson on July 29, 2003 at 08:34 PM EDT #
Posted by Gavin on July 29, 2003 at 11:50 PM EDT #
Posted by Nadeem on July 30, 2003 at 07:35 PM EDT #
Posted by Gavin on July 31, 2003 at 05:45 AM EDT #
Posted by Nadeem on July 31, 2003 at 06:30 AM EDT #
Posted by Gavin on July 31, 2003 at 02:50 PM EDT #