« More JSP best practi... | Main | Relegated to the... »

History of Roller performance or lack thereof.

As you may or may not remember, I started developing Roller, in part, to teach myself about database driven web applications in the world of Java and J2EE. So, the history of Roller performance is also the history of my school of hard knocks education in Java webapp performance. I'm not sure if anybody can learn anything of value from this, but I feel like reviewing the story, and this is my blog, so that's what I'm gonna do. If you don't like it, you know where the back button is. Here is the story so far.

In the beginning, I implemented Roller using EJB. Every Roller business object was an Entity Bean and every Roller business tier interface was a Session Bean. I learned a lot about EJBs and XDoclet and everything was just fine, but Roller, or Homeport as I used to call it, took 20 seconds to render a weblog page. I studied the various EJB patterns and realized that returning a collection of Entity Beans from the business tier was not a good idea. I learned about the Data Access Object and Value Object patterns. I discovered that XDoclet could generate Value Objects (which Sun now calls Data Transfer Objects) for me, and I changed the business tier interfaces to return the light-weight data objects rather than Entity Beans. Much better! That was performance enhancement #1.

Once I had the business tier interfaces returning data objects, I found that it was easy to convert Roller over to use Castor for persistence. I was tired of waiting for EJB containers to start, tired of waiting for EJBs to deploy, and generally disillusioned with EJB. So, I created Castor mappings to map the XDoclet generated Value Objects to database tables and switched over to Castor. Again, much better. That was performance enhancement #2.

When I tried to get Roller to display my bookmarks page, which has a hundred or more bookmarks, I still found that Roller was way too slow. Rendering a page was taking 10 seconds or so. I realized that I was not taking advantage of database connection pooling. Configuring database connection pooling was performance enhancement #3.

I'm not sure about the next chapter, but I believe that it happened this way. I was forced to switch from Resin to Tomcat and Tomcat was slow (back then it was slow, now it screams ;-). I found that OSCache could allow me to hide, almost completely, this problem. OSCache was enhancement #4.

I'm a little fuzzy about this one too, but here is what I remember. One day I read that some guy named Anthony Eden had decided to set up Roller and to offer free accounts on a site called FreeRoller. I figured he would close FreeRoller off after reaching 10 or 20 users. He didn't. He kept going and going. Now I'm not sure if he is ever going to stop. Anyhow, Roller started running out of memory, so we adjusted to session timeout to a lower value and I moved the Roller business tier implementation from being a session thing to being an application thing. Those changes, call them enhancement #5, fixed the out-of-memory problems.

FreeRoller did pretty well for a while, especially considering the el-cheapo little eMachines box that Anthony started with. Eventually, it got bogged down again. I was starting to think that our persistence framework was at fault and I had heard that Hibernate was much faster than Castor, so I switched to Hibernate. Hibernate seemed to be about twice as fast. I'll call that enhancement #6.

Around this time, JavaLobby offered to host FreeRoller. FreeRoller moved from Anthony's little eMachines box to a big honkin' JavaLobby server. Everybody danced a wild fandango of glee and revelled in the new found speed and snappiness of Roller. FreeRoller != Roller, but for the sake of this story, we'll call that enhancement #7. The fast machine worked great for a while, but eventually it too started to slow down. At first, users did not notice the problem, but Rick Ross and Matthew Schmitt complained that Roller's load was way too high. After some investigation, we figured out that the RSS Servlet was doing more work than it should. So, enhancement #8 was to make sure that RSS feeds are cached and that the IF-MODIFIED-SINCE header was being handled and handled efficiently. That fixe brought the Roller load down to reasonable levels.

That brings us to today. The number of FreeRoller users and readers has grown and FreeRoller is again creaking and straining under the pressure. I hadn't noticed this because I was so busy with other things, but Anthony noticed, and so did Lance. Lance did some research and theorized that the problem is that Roller uses too many database connection to process each request. Each Roller weblog page is rendered by a user-edited Velocity script with a dozen or more embedded macros. Each of these macros makes one or more calls to the Roller business tier and each Roller business tier method opens a connection, performs some database operation, and closes the connection. Roller uses database connection pooling, but even so, grabbing and releasing a connection dozens of time for each request can't be a good thing.

Long story long, Lance modified Roller to use one connection per request. Now we are testing Roller and if all goes well, I'll backport the changes to the Roller 0.9.7 branch so that they can easily be applied to FreeRoller this weekend. I'm keeping my fingers crossed and hoping that enhancement #9 will hold us over for a couple of months while we figure out enhancement #10. I've learned a lot from Roller and I suspect that I have a lot more to learn.

Comments:

Thank you, but I cannot take the credit for that realization; someone on the list pointed it out (in fact, I believe a couple people did). If I wasn't such a lazy so-and-so I'd go look up the messages in the archive - but you know who you are. Will the real too-many-db-conx's people please stand up.

Posted by Lance on July 29, 2003 at 12:02 PM EDT #

How radical did #9 this affect the overall design? Did this affect the way hibernate was used? Any tips on not making the same mistake? Matt

Posted by Matt P on July 29, 2003 at 12:44 PM EDT #

Uups, definitly too much medicine for patient. Maybe try something homeopathic and get rid of this patterns :->.

Posted by Gerhard on July 29, 2003 at 01:16 PM EDT #

Question: Why were you "forced" to move from Resin to Tomcat?? I used Resin a lot early last year and was "forced" to move to Tomcat for department politics reasons, but technically speaking I can't think of anything that would force a move. Caucho did an excellent job with the speedy Resin, and they were always responsive when contacting them regarding bugs or new features. Just curious....

Posted by Gerry on July 29, 2003 at 02:20 PM EDT #

This was over a year ago. I was forced to move to Tomcat because I switched ISPs and my new ISP did not support Resin.

Posted by Dave Johnson on July 29, 2003 at 02:36 PM EDT #

For what its worth I found the problems with the problematic use of Hibernate sessions. I have already spoken to Dave about getting write access to the Roller CVS so perhaps the next time something like this occurs I will just fix the darn thing myself. ;-)

Posted by Anthony Eden on July 29, 2003 at 02:42 PM EDT #

"problems with the problematic use" - what was I thinking?

Posted by Anthony Eden on July 29, 2003 at 02:43 PM EDT #

I've been working on how Sessions are used, and came close to having it last night. I may very well have something to checkin tonight.

Posted by Lance on July 29, 2003 at 03:36 PM EDT #

I have to disagree with you when you consider switching from EJB to Castor as a performance enhancement. A WELL designed EJB system perform and scale as good or better than the same system that uses Hibernate for persistence.

Posted by Nadeem Bitar on July 29, 2003 at 06:28 PM EDT #

Nadeem, can you back that up with some facts? I'd love to see that... of course, I don't believe it, but I'd love to see it.

Posted by Jason Carreira on July 29, 2003 at 07:24 PM EDT #

Jason how about you proving the contrary with a real world application that uses transactions, security, stateful sessions (for some use cases), asynchronous processing ... I am interested in scalability and performance. I am actually toying with the idea of porting a hibernate based app to using MDB + CMP and perform extensive benchmarks unfortunately my schedule is saturated right now.

Posted by Nadeem Bitar on July 29, 2003 at 07:51 PM EDT #

Nadeem, you may be right. I don't know. I was turned off by EJBs because of the extra work and development-time hassle they create (deployment descriptors, container overhead, learning curve). From my point of view this was a performance enhancement, but now that Roller is deployed on site with heavy traffic, EJB might again make sense. We have a Castor and a Hibernate implementation of the Roller business tier. Want to do an EJB implementation? ;-)

Posted by Dave Johnson on July 29, 2003 at 08:34 PM EDT #

Guys, opening multiple Hibernate Sessions / connections is a definite performance antipattern, but fortunately the ThreadLocalSession pattern makes it VERY easy to fix. Check out our Wiki; you'll have Roller MUCH faster with just a little bit of work. >> WELL designed EJB system perform and scale as good or better than the same system that uses Hibernate for persistence. << Rubbish. This is simply incorrect. I have an inbox full of people who have ported from Entity Beans to Hibernate and seen enormous performance increases. I could explain in detail the technical reasons for this - but it won't fit in a comment and I'm currently writing a book about it. A *real* ORM solution like TopLink or Hibernate will outperform and probably outscale entity beans any day of the week!

Posted by Gavin on July 29, 2003 at 11:50 PM EDT #

Gavin, I would be very interested to see how hibernate scale btter than CMP. Hibernate uses reflection extensively and even with the most modern JVM that would cause a performance hit on high loads.

Posted by Nadeem on July 30, 2003 at 07:35 PM EDT #

>> Gavin, I would be very interested to see how hibernate scale btter than CMP << Then write a Hibernate application and Watch It Happen. duh. >> Hibernate uses reflection extensively and even with the most modern JVM that would cause a performance hit on high loads << umm, no it doesn't ... it uses runtime bytecode generation now, which is just as fast as EJB-style sourcecode generation. But anyway, that has absolutely NOTHING to do with *scalability*! Do you know what "scalability" means?? Go look up a definition of the word before posting again, please. "More scalable" absolutely does NOT mean "faster". Of course, Hibernate is also faster in absolute terms, as reported by many, many users. Why? Because it accesses data more efficiently than CMP engines. ie. there is less disk access / network traffic.

Posted by Gavin on July 31, 2003 at 05:45 AM EDT #

Gavin If you reread my comment i never claimed that reflection has anything to do with scalability i just wrote that it would impact performance on. I resent your provocative and childish comments, grow up. "There is less disk access /network traffic" Have you heard of local interfaces? "Hibernate is also faster in absolute terms as reported by many, many users" I bet your users are benchmarking their apps and sending you the results .

Posted by Nadeem on July 31, 2003 at 06:30 AM EDT #

>> Have you heard of local interfaces? << And how do local interfaces help reduce the extra network traffic involved in inefficient data access. >> grow up << ok. >> I bet your users are benchmarking their apps and sending you the results << correct.

Posted by Gavin on July 31, 2003 at 02:50 PM EDT #

Post a Comment:
  • HTML Syntax: NOT allowed

« More JSP best practi... | Main | Relegated to the... »

Welcome

This is just one entry in the weblog Blogging Roller. You may want to visit the main page of the weblog

Related entries

Below are the most recent entries in the category Roller, some may be related to this entry.