Blogging Roller

I upgraded FreeRoller to Roller 0.9.8-dev last night. The upgrade went well. FreeRoller is noticably faster and, more importantly, FreeRoller made it through the night. There are a lot of fixes in Roller 0.9.8 as you can see from the Roller <a href= "http://opensource.atlassian.com/projects/roller/secure/BrowseProject.jspa?id=10000&report=roadmap"> roadmap, but Lowem has found that there are also gotchas for folks using "undocumented" features. If Roller 0.9.8-dev continues to work well, I'll release it as Roller 0.9.8 this weekend.

will be <a href= "http://freeroller.net/page/davetest">blogging FreeRoller planned downtime, system restarts, etc.

Now that the guy who started FreeRoller has bailed out, I'm going to start take a little more active role in FreeRoller administration. I don't really want to be the FreeRoller administrator (Roller is my baby not FreeRoller), but I don't want to see a thousand (1507 at last count) bloggers ejected from the blogosphere either. Plus, I'm interested in solving the FreeRoller performance problems or at least understanding them.

What's the plan? I talked to the folks at JavaLobby and we agreed on a couple of things: 1) new user registration will be closed until we get a better handle on performance and 2) FreeRoller will be moving to the improved Roller 0.9.8 codebase as soon as possible.

What about the performance problems? Gavin King of Hibernate is giving us some advice on using some of Hibernate's performance features, which I will deploy (after some testing/benchmarking) to FreeRoller with the Roller 0.9.8 code base. Since Roller uses page caching so heavily, I think it is a little unlikely that the persistence layer is to blame, in fact it is more likely that OSCache is to blame (or, of course, Roller itself), but at this point I'm willing to try anything.

Want to help? Grab the Roller source and have at it.

Anthony bailed out, leaving FreeRoller without an administrator.

<a href= "http://brainopolis.dnsalias.com/roller/page/lance/20030903#circle_blog">Lance: We continue working to make Roller faster, and to make Roller better. Some people see Roller as a product, me, I see it as a hobby. Everybody wants to be better at their hobby.

Lance has a new baby boy and a new open source project. He hasn't posted any baby pictures yet so I don't know what his new baby looks like, but his new open source project, Atom4J, just landed in the Roller CVS and I'm reviewing the code now.

Atom4J is a Java implementation of Atom, everybody's favorite new weblog syndication format and API. Instead of simply adding Atom support into Roller, Lance chose a more difficult and potentially more rewarding path. He released Atom4J as a standalone product that can be used to add Atom API support to any Java-based weblogging system. This is a Good Thing™. You can find out more and download Atom4J at the OSJava.org site.

I finally got around to uploading my new theme. It's called Bluebar.

Courtesy of <a href= "http://gianfvil.gamersrevolt.it/Zimbabwe/Lilac-breasted%20roller%202.htm">Guiliano and Silvio.

Everybody has trouble installing Roller and it's really not Roller's fault. I lay all of the blame on DBCP. That's a lame excuse, I know. Anyhow... Min has a plan to build an awesome Roller installer. Go Min!

And, speaking of Blojsom: Blojsom 2.0 was released today. Coincidentally, Blojsom co-developer David Czarnecki bumped up his revision number too. Happy Birthday David!

A number of Roller performance/load fixes have been applied to FreeRoller over the last month or so including: persistence session per request, upgrade to OSCache 2.0-beta2, enabling OSCache unlimited disk cache, and upping the page cache timeouts from 1 to 3 hours. The unlimited disk cache was enabled this morning and the performance and load improvements seem to be pretty significant, but there are still reports of some 404 errors that need to be investigated.

If you display referers on your site, you'll notice the higher page cache timeouts because your referers list will only be refreshed every three hours (unless you post or manually flush your cache).

The past couple of days, I have noticed that some little twerp has been changing random words in the Roller wiki. I'm getting tired of making little corrections to the wiki, so I've turned security back on. The Roller developers have user name and passwords for the wiki. If you want one too, let me know. I might grant you one. I know, I know! it's not the wiki way. It's my way - or the highway! Mama Musings was right all along. Foo!

I've been working on improving the performance of the Roller persistence tier. I've been doing this work in the old Roller 0.9.7 branch to make it easy for FreeRoller to deploy the improvements. Here are some details on the improvements.

Roller uses an O/R mapping framework persist objects in a relational database. Before the improvements, every call into the Roller persistence layer would open, use, and close a persistence session (either a Hibernate Session or a Castor Database). Roller often makes multiple persistence layer calls to process a web request and, so, Roller would often open, use, and close multiple persistence sessions per web request. A number of people pointed out that this is a bad thing.

To fix this problem, I changed Roller to ensure that each incoming web request gets it's very own persistence session and all persistence layer calls made on behalf of that request use that same persistence session. That means that there is, at most, one persistence session per incoming request. Persistence sessions don't hang around, each one lasts for the duration of one web request only.

These change improved Roller's performance, but not by much. Here are the JMeter test results. As always, comments, suggestions, and criticisms welcome. I think it is interesting that Resin performed better than Tomcat on the baseline test, but not the Roller test (a sign that I screwed something up?).

Update: FreeRoller is deploying these changes now. Even though the performance gains are minimal, I'm hoping the load and the number of MySQL threads will be reduced. I'll be playing with Hibernate caching and the new OSCache beta release tonight to see if they can offer any additional improvements.

Big round of applause for Anthony Eden and Min Idzelis who have just joined the Roller project. Hip, hip, hoooooray! By the way, Min lives right in the Triangle. I'll have to convince him to come out to the next RTP Bloggers lunch.

Charles Miller - Version Numbers and You: Too many Open Source projects treat Version 1.0 as some kind of Holy Grail that can only be reached when the project is perfect. I find that highly annoying, because it makes it really, really difficult to tell a sketchy alpha from production code that is just still in pre-1.0 because the author wants it to do everything.

If we add multiple categories per post, I'd be happy calling the next release Roller 1.0 rather than Roller 0.9.8.

Roller has been honored by the BileBlog. Most of Hani's criticisms are valid, but none new or especially insightful, at least not to me. I'm a little surprised he did not attack the sucky UI as it is a prime target. Also, it is interesting that he zoomed in on the multiple connections per request problem, I think he had an inside tip ;-) That is what we are working on this week. I sincerely hope he is correct and that fixing the multiple connection per request (enhancement #9 from my earlier post) solves the FreeRoller current performance problems.

As you may or may not remember, I started developing Roller, in part, to teach myself about database driven web applications in the world of Java and J2EE. So, the history of Roller performance is also the history of my school of hard knocks education in Java webapp performance. I'm not sure if anybody can learn anything of value from this, but I feel like reviewing the story, and this is my blog, so that's what I'm gonna do. If you don't like it, you know where the back button is. Here is the story so far.

In the beginning, I implemented Roller using EJB. Every Roller business object was an Entity Bean and every Roller business tier interface was a Session Bean. I learned a lot about EJBs and XDoclet and everything was just fine, but Roller, or Homeport as I used to call it, took 20 seconds to render a weblog page. I studied the various EJB patterns and realized that returning a collection of Entity Beans from the business tier was not a good idea. I learned about the Data Access Object and Value Object patterns. I discovered that XDoclet could generate Value Objects (which Sun now calls Data Transfer Objects) for me, and I changed the business tier interfaces to return the light-weight data objects rather than Entity Beans. Much better! That was performance enhancement #1.

Once I had the business tier interfaces returning data objects, I found that it was easy to convert Roller over to use Castor for persistence. I was tired of waiting for EJB containers to start, tired of waiting for EJBs to deploy, and generally disillusioned with EJB. So, I created Castor mappings to map the XDoclet generated Value Objects to database tables and switched over to Castor. Again, much better. That was performance enhancement #2.

When I tried to get Roller to display my bookmarks page, which has a hundred or more bookmarks, I still found that Roller was way too slow. Rendering a page was taking 10 seconds or so. I realized that I was not taking advantage of database connection pooling. Configuring database connection pooling was performance enhancement #3.

I'm not sure about the next chapter, but I believe that it happened this way. I was forced to switch from Resin to Tomcat and Tomcat was slow (back then it was slow, now it screams ;-). I found that OSCache could allow me to hide, almost completely, this problem. OSCache was enhancement #4.

I'm a little fuzzy about this one too, but here is what I remember. One day I read that some guy named Anthony Eden had decided to set up Roller and to offer free accounts on a site called FreeRoller. I figured he would close FreeRoller off after reaching 10 or 20 users. He didn't. He kept going and going. Now I'm not sure if he is ever going to stop. Anyhow, Roller started running out of memory, so we adjusted to session timeout to a lower value and I moved the Roller business tier implementation from being a session thing to being an application thing. Those changes, call them enhancement #5, fixed the out-of-memory problems.

FreeRoller did pretty well for a while, especially considering the el-cheapo little eMachines box that Anthony started with. Eventually, it got bogged down again. I was starting to think that our persistence framework was at fault and I had heard that Hibernate was much faster than Castor, so I switched to Hibernate. Hibernate seemed to be about twice as fast. I'll call that enhancement #6.

Around this time, JavaLobby offered to host FreeRoller. FreeRoller moved from Anthony's little eMachines box to a big honkin' JavaLobby server. Everybody danced a wild fandango of glee and revelled in the new found speed and snappiness of Roller. FreeRoller != Roller, but for the sake of this story, we'll call that enhancement #7. The fast machine worked great for a while, but eventually it too started to slow down. At first, users did not notice the problem, but Rick Ross and Matthew Schmitt complained that Roller's load was way too high. After some investigation, we figured out that the RSS Servlet was doing more work than it should. So, enhancement #8 was to make sure that RSS feeds are cached and that the IF-MODIFIED-SINCE header was being handled and handled efficiently. That fixe brought the Roller load down to reasonable levels.

That brings us to today. The number of FreeRoller users and readers has grown and FreeRoller is again creaking and straining under the pressure. I hadn't noticed this because I was so busy with other things, but Anthony noticed, and so did Lance. Lance did some research and theorized that the problem is that Roller uses too many database connection to process each request. Each Roller weblog page is rendered by a user-edited Velocity script with a dozen or more embedded macros. Each of these macros makes one or more calls to the Roller business tier and each Roller business tier method opens a connection, performs some database operation, and closes the connection. Roller uses database connection pooling, but even so, grabbing and releasing a connection dozens of time for each request can't be a good thing.

Long story long, Lance modified Roller to use one connection per request. Now we are testing Roller and if all goes well, I'll backport the changes to the Roller 0.9.7 branch so that they can easily be applied to FreeRoller this weekend. I'm keeping my fingers crossed and hoping that enhancement #9 will hold us over for a couple of months while we figure out enhancement #10. I've learned a lot from Roller and I suspect that I have a lot more to learn.

I've been too busy do anything other than watch the emails fly by, but Min has been busy. He added full weblog search to Roller via Lucene. Take a look at Matt's write-up for details.

Good stuff from Mr. Lowem: <a href= "http://freeroller.net/page/lowem/20030718#more_roller_annoyances">Minor Roller anoyances and <a href= "http://freeroller.net/page/lowem/20030718#more_roller_annoyances">More Roller Annoyances. Interested in improving Roller? Join us on the Roller development mailing list. There are lots of little bugs to fix and minor improvements to be made, great for those getting started with server-side Java, but there are probably some deeper architectural issues for those interested in performance and scalability.

It's been more than a month since I refreshed the code running behind this site. I'm now running the latest Roller 0.9.8-dev from CVS. This means I'm on Struts 1.1 final (thanks to <a href= "http://brainopolis.dnsalias.com/roller/page/lance/20030702#upgrade_roller_to_use_struts"> Lance) and Hibernate 2.0.1 (thanks to the Hibernate guys). I'm hoping that I might actually get some time to work on Roller in the next couple of weeks. I need to. The natives are getting restless, but I'm going to the beach and you can't stop me.

<a href= "http://brainopolis.dnsalias.com/roller/page/lance/20030702#upgrade_roller_to_use_struts"> Lance has moved the Roller codebase forward and has cross over into Struts 1.1 land. Hurrah! That reminds me... I really need to get the latest Roller 0.9.8-dev running on my site so that I can apply the RSS performance enhancements that I made to Roller 0.9.7 for FreeRoller.