Blogging Roller

Anthony reports that FreeRoller is getting a new home, thanks to the JavaLobby.

Anthony Eden: FreeRoller is going to have a new home. The JavaLobby has offered to host FreeRoller on one of their servers which is significantly more powerful than "el crappo", the dinky box which currently hosts FreeRoller [more...]

I just uploaded Roller 0.9.6.5 to SourceForge. This is a minor "patch" release that includes a web.xml bug fix and some performance enhancements including: simplified main page, better last-update query, RSS caching, etc. This is not the long awaited <a href= "http://opensource.atlassian.com/projects/roller/IssueNavigator.jspa?reset=true&pid=10000&fixfor=10010">Roller 0.9.7 release with support for comments. If you want that you'll have to get it from CVS or wait until the third week of April.

In the evenings, I've been making progress on the Hibernate impementation of the Roller business tier (a.k.a backend) and I completed a Hibernate version of NewsfeedManager, but I keep getting bogged down in refactoring. I can't resist making some improvements to the Roller object model and when I make improvements in the objects, I have to make the corresponding changes in Castor. Then I get diverted into refeactoring the Castor implementation.

This is the drawback of having JUnit tests in place: it makes refactoring so easy that you do it just because you can. Sometimes, I'm not sure if I'm making useful changes or just macdinking. The only think that is holding me back is the lack of unit tests in the Roller presentation tier.

I need to put my nose to the grindstone and wrap this stuff up. I want to get back to the cool stuff, like fixing Roller's comments features and stealing <a href= "http://www.blojsom.com/blog/blojsom/?permalink=A480AB5244E2F1A1656518EA87A4801C.txt">Trackback and <a href= "http://www.blojsom.com/blog/blojsom/?permalink=blojsom-metaweblog.html">MetaWeblog API support from Blojsom.

I tried the kill -QUIT trick on FreeRoller this afternoon after it locked up. Looking at the stack traces for Tomcat's 75 threads, I found that most were either waiting for database connections or involved reading cache entries from the disk.

Recently, I had changed to OSCache settings so that Roller caches to disk rather than to memory. I did that because Roller was running out of memory. Caching to disk fixed the out-of-memory problem, but at the expense of heavy disk access. So, I changed the OSCache settings back to cache to both disk and to memory. OSCache seems to have some limitations in this area. Apparently, if you configure OSCache to cache in memory and to disk, you can limit the number of entries in the memory cache, but the disk cache is unlimited - which is not the best situation.

Also, I changed the DBCP connection pooling parameters to reduce the maxIdle and maxWait so that threads will timeout quickly rather than hanging around waiting for connections and thus forcing the creation of new threads.

Those changes kept FreeRoller up for a much longer period of time, but did not fix the problem.

Andy Oliver: I followed the instructions, step by step, substituting the postgresql script for the mysql and the driver name, etc. (Take for granted that I approximately know how to do these things since I do Java/database-related apps for a living) [...] Nope...It doesn't look like it worked...

Andy Oliver: competence is required to judge competency and that the incompetent often don't know that they are.

I'd like to point out that we have this thing, we call it a "<a href= "http://sourceforge.net/mail/?group_id=47722">mailing list," where we answer installation questions and otherwise help people who are trying to get started with Roller. Look into it and if you have any suggestions for making the Roller install process better, please share them with us.

FreeRoller keeps on running out of threads to process incoming requests. This could be a problem in Roller, but I suspect that we may have run up against this bug in the Tomcat 4.0.X HTTP connector:

BUG5735: HTTP connector running out of processors under heavy load

The bug is marked as fixed, but I think that means the fix is in the new Tomcat 4.1.X Coyote HTTP connections.

Next time this happens, I'll try Glen Nielson's advice (from the bug report):

I would recommend that you dump the stack for all running threads when
you experience this problem.  This can help identify what is causing the
problem. By reviewing the stack dump for each thread you can determine
whether the problem is due to Tomcat or your application.

On unix you do a kill -QUIT {tomcat java pid) to cause the thread stack's 
to be dumped to catalina .out.

A Processor for Tomcat runs your application code, delays in your code
can cause additional processor threads to be created to handle new requests.
Possible application or configuration problems which can delay requests:

   Connection delays due to networked services such as a db.
   Connection delays due to running out of pooled resources.
   Thread synchronization deadlocks.
   A cascading affect where many new processors get created due to
   excessively long JVM Garbage Collections.  start java with
   -verbose:gc to detect this.

Cameron Purdy commented on FreeRoller performance recently:

Apparently, over half the load is on the RSS/XML side (not the "interactive web/HTML" side,) and it's enough to completely saturate their T1 connection. Wow! And it's running on an eMachine (a US$200 "Walmart computer") that was given up after it had served its useful life. So in a way, it's amazing that it runs at all. However, Anthony has plans to cluster it, and with a couple of commodity servers to run on (plus enough bandwidth?), it should be back flying again, hopefully before too long.

Recently, I've been helping Anthony Eden keep FreeRoller up and I've been watching the logs and the stats. Cameron is right, FreeRoller gets a hell of a lot of RSS traffic and FreeRoller is running on a pretty light-weight machine: a 450Mhz Pentium II with 256MB RAM.  This might be a fine an Apache/mod_perl app, but for big beefy Roller/Tomcat/Struts/Velocity/Castor it's just not enough.

Still, I'm doing what I can to improve performance. For example: this weekend I added caching to FreeRoller's RSS feed. To do this I had to add: 1) handling for the IF-MODIFIED-SINCE header in the PageCache ServletFilter and 2) a new Filter Mapping to map '/rss/*' to Roller's OSCache based PageCache Servlet Filter. This should speed average RSS response time, reduce memory usage, and database access. It may have some effect on the FreeRoller RSS feeds, so if you notice a problem in a FreeRoller RSS feed let me know about it.

I developed Roller without unit testing, but I'm not going proceed with the Hibernate implementation without having tests in place. Last night, I checked in tests for the Roller NewsfeedManager, the simplest of the Roller backend managers. The next step is to write or generate Hibernate mappings for the NewsfeedData object and code up a new NewsfeedManagerImpl.

While I was working on that, Matt implemented remember me, added email notification for comments, and upgraded his site to the latest Roller from CVS (not even I am ready to do that ;-)

Finally, I'm ready to start working on the Hibernate implementation of the Roller Weblogger backend. I ended up doing a lot more refactoring than I had intended. I told you about how I moved code from the Castor implementation into abstract base classes that can be used by the Hibernate implementation. Now I'll describe the changes that I made in the Roller build and code-generation process.

In the original Roller build process, illustrated in Figure 4 of the article Building a J2EE Weblogger, I used abstract javax.ejb.EntityBean classes as the meta-data basis for generating code via the XDoclet EJBDoclet task. I was subverting EJBDoclet: using it to generate Data Objects, Struts Forms, and Castor mappings but not using any of it's EJB output.

That worked pretty well, but eventually it became a problem. The generated Data Objects were just dumb data-holders and, over time, we realized that they need to be smarter "business objects." The Data Objects were regenerated on every build and that made it diffucult, if not impossible, to add new methods, new logic, and new collections.

The new Roller build/code-gen process

<img src="http://www.rollerweblogger.org/resources/roller/xdoclet-roller-sm.jpg" alt="Diagram of Roller build process">

The new Roller build process, or at least the code-generation part of it, is shown above. We now start with some hand written "Plain Old Java objects" or POJOs. We still have to subvert EJBDoclet because the XDoclet <strutsform> and <castormapping> can only exist inside EJBDoclet.

I had to use Matt Raible's patched version of <strutsform> (from his struts-resume example) because the one in XDoclet 1.2b2 works only if the source class extends javax.ejb.EntityBean and the new Roller POJO's don't do that. Matt and I consider this to be a bug in EJBDoclet, but I'm not sure the XDoclet guys agree. Maybe Roller should define it's own <strutsform> that works on any POJO and that inserts validator tags (something Matt also added in his struts-resume example).

The Roller build/code-gen process is still not perfect, but it's "good enough" for me to begin my Hibernate implementation of the Roller Business Tier interfaces. I'll be blogging as I go so stay tuned.

Matt Raible has successfully tested Russell Beattie's <a href= "http://www.russellbeattie.com/notebook/20030225.html#231612">world-famous <a href= "http://www.manywhere.com/Moblogger.html">ManyWhere Moblogger with Roller.

To prepare for the move to <a href= "http://hibernate.bluemars.net">Hibernate, I've been refactoring the current implementation of the Roller business tier. There is a lot of code in the Castor implementation of the <a href= "http://www.rollerweblogger.org/javadoc/org/roller/model/package-summary.html">persistence manager interfaces that is generic enough to be used in the Hibernate impementation. So, I'm moving that code up into abstract base classes that may be used by both implementations.

As FreeRoller users and readers know, FreeRoller is having problems keeping up with the load of 500+ weblogs and the 101-list traffic. Sometimes the main page takes several minutes to load and often the system just collapses under the stress. According to Anthony Eden, who runs FreeRoller, the system is powered by one little eMachines box. I could use that as an excuse, but, regardless of the hardware, the poor performance of FreeRoller reflects poorly on the Roller software.

I considered using a profiler to figure out the root cause of the performance problem, but a profiler is difficult to setup for a Servlet app, there's a learning curve, and there are only so many Roller hours in my weekend. Plus, I had a gut feeling that the Roller main page, index.jsp, was causing the slow-downs.

Looking at the code behind index.jsp, I found that it uses several queries to build a list of weblogs, hit counts, and the last update time for each weblog. The queries are not complex, but because they are executed using an Object-Relational (O/R) persistence framework (Castor) each query results in the creation of one object per row returned. FreeRoller has over 500 weblogs, so each refresh of the index.jsp page results in the creation of thousands of objects. Even with page caching in place to reduce the frequency of refreshes, this has the potential to bog down the whole system.

I rewrote the code behind index.jsp to use only one query and I also added a limit to that query so that a Roller administrator can limit the number of weblogs displayed on the main page. Next, I did some load testing with JMeter. I set up a JMeter test plan to use 3 threads to hit index.jsp, a 20-second ramp up, and a 300 ms delay between requests. I did this testing on a 1.5Ghz Athlon, 768MB RAM box running Windows XP, Java 1.4.1, and Tomcat 4.1.18. Here are the results:

                   Avg.  Dev.  Throughput  Note

1) Old index.jsp 1051ms  8629  120/min     Heavy CPU usage
2) New index.jsp  248ms  1326  248/min     Limit 500 weblogs
3) New index.jsp   93ms   186  247/min     Limit 30 weblogs
4) TC index.jsp    90ms    72  337/min     Tomcat example page

The old index.jsp would peg the CPU at 100% for minutes at a time during cache refreshes. I'm not sure why, but I suspect that multiple requests were simultaneously refreshing the cache (I'm not sure how OSCache hanldles this). During test #1, Roller was so slow as to be almost unusable. Test #2 and #3 show that the new index.jsp is a great improvement. With a limit of 500 weblogs, the CPU usage is very heavy during a cache refresh but doesn't get to 100%. With a limit of 30 weblogs, the cache refresh does not have a noticable effect on the CPU. Test #4 tests the Tomcat examples page, it is included only as a baseline for comparison.

I made the above changes in the Roller 0.9.6 code branch so that they can me applied to FreeRolller right away. I'm still not happy about Roller performance and memory usage. I'm sure there is plenty of room for improvement. My next experiment is implementing the Roller business tier with the Hibernate O/R framework. I'm curious to see how it compares to Castor.

Lance is experimenting with Prevayler by using it and JXPath to implement the Roller business tier. Cool stuff. I'm gearing up for a Hibernate implementation; more about that later.

Roller uses the Castor persistence framework. FreeRoller can be very slow.

Are the two things related? I have no real empirical evidence (yet) to prove a link, but I did find an interesting link of a different sort in my referer logs this morning to a blog entry by (Roller user) Matthew Porter. Back in December 2002, Matthew chose Hibernate instead of Castor for the Java Lobby Community Port (JLCP) project. Here's why:

Persistent Framework Choice for JLCP: Castor was not chosen for two primary reasons. The first is the lack of development of Castor in the past year. In addition, one the tests we performed at DMI, Castor was significantly slower than other PFs- to the point where it was intolerable. The first reason and recent tests led me to believe that the situation regarding speed had not changed.

Also, an interesting link was posted in a comment on my Long Transactions post yesterday. This is a pretty interesting article:

O/R Mapping with Castor JDO in the Real World: Castor holds up to its promises in simple testing and trial runs. However, it has proven to fall short in some practical issues with our application of about twenty-five data classes and as many tables. Most of our problems come from the need to hold onto objects across transactions and perform complex updates.

I'm tempted to rewrite the Roller backend using Hibernate just for the hell of it, but I really should to do some profiling of Roller to see where the problem lies, don't you think? I guess I could do a 30-day eval of OptimizeIT or JProbe, but I would be happy to hear your recommendations for free and/or open source profiling tools. Got any?

I upgraded the Roller main branch (and this site) to Struts 1.1b3, Castor 0.9.4.3, Xerces 2.3, and Ant 1.5.1. I'll pop in Struts 1.1 RC1 when it arrives. I think we are pretty close to a Roller 0.9.7 release. Lance has finished replacing all of the old Roller "macros" with VelociMacros (don't worry, we'll keep the old macros around for a while) and we only have a couple of open issues left to go. I do want to do a little work on the comments feature before we release. Lance did a great job on comments, but I want to polish it up a little before it goes out.

I'm wondering the same thing. If they were running in the same ServletContext, we might be able to make them do some interesting tricks.

Matt Croydon: How easy would it be to integrate something like JSPWiki with Roller? I really like the integration that Mr. Orchard has accomplished over at 0xDECAFBAD. I'm not personally a Roller user (tho I've played with it and love it), but given the JSP connection, I think it might not be too impossible.

It's nice that there are multiple competing Roller sites now.

<a href= "http://blogs.application-servers.com/blogs/page/leecho/20030131#sorry_blog_moved">Carlos Villela: I had to move this blog to another Roller. Sorry guys and gals, but freeroller.net is just too slow, and I don't know if it's a problem in their bandwidth, mine, or if they're just too loaded.

This release is the same code-base that Anthony Eden is currently running on FreeRoller. It includes some minor bug fixes and makes more effective use of caching on the main (index.jsp) page. Get it on SourceForge. Roller 0.9.7 is still a little ways off. By the way, FreeRoller is processing 1,000,000 hits per month, all on a "dinky" little eMachines box.

I had to laugh when I saw the post on Matt Croydon's site, especially since it was my fault. The basic problem here is that Roller weblog entries are run through Velocity before they are displayed. I added this feature so that user's could use Roller macros within weblog posts to do things like displaying images. Now I'm thinking that Velocity is overkill. Velocity is perfect for the Roller page templates, but for the weblog posts themselves? Lance's "hotwords" idea is probably more appropriate.

Lance has resolved his Velocity difficulties and is well on the way to providing a full set of Velocimacros for Roller. This will make it much easier for Roller administrators and users to customize Roller. It will also bring Roller in line with Velocity best practices.