Blogging Roller

eclipse.exe -vmargs -Xverify:none -XX:+UseParallelGC -XX:PermSize=20M  
 -XX:MaxNewSize=32M -XX:NewSize=32M -Xmx96m -Xms96m

Sosume applies some new GC configuration settings, written about in JavaWorld and recommended for Netbeans to improve Eclipse performance in his post: Improving Eclipse performance by using vm arguments. I'm trying these settings myself now, but with -Xmx256m.

I tried the kill -QUIT trick on FreeRoller this afternoon after it locked up. Looking at the stack traces for Tomcat's 75 threads, I found that most were either waiting for database connections or involved reading cache entries from the disk.

Recently, I had changed to OSCache settings so that Roller caches to disk rather than to memory. I did that because Roller was running out of memory. Caching to disk fixed the out-of-memory problem, but at the expense of heavy disk access. So, I changed the OSCache settings back to cache to both disk and to memory. OSCache seems to have some limitations in this area. Apparently, if you configure OSCache to cache in memory and to disk, you can limit the number of entries in the memory cache, but the disk cache is unlimited - which is not the best situation.

Also, I changed the DBCP connection pooling parameters to reduce the maxIdle and maxWait so that threads will timeout quickly rather than hanging around waiting for connections and thus forcing the creation of new threads.

Those changes kept FreeRoller up for a much longer period of time, but did not fix the problem.

Andy Oliver: I followed the instructions, step by step, substituting the postgresql script for the mysql and the driver name, etc. (Take for granted that I approximately know how to do these things since I do Java/database-related apps for a living) [...] Nope...It doesn't look like it worked...

Andy Oliver: competence is required to judge competency and that the incompetent often don't know that they are.

I'd like to point out that we have this thing, we call it a "<a href= "http://sourceforge.net/mail/?group_id=47722">mailing list," where we answer installation questions and otherwise help people who are trying to get started with Roller. Look into it and if you have any suggestions for making the Roller install process better, please share them with us.

FreeRoller keeps on running out of threads to process incoming requests. This could be a problem in Roller, but I suspect that we may have run up against this bug in the Tomcat 4.0.X HTTP connector:

BUG5735: HTTP connector running out of processors under heavy load

The bug is marked as fixed, but I think that means the fix is in the new Tomcat 4.1.X Coyote HTTP connections.

Next time this happens, I'll try Glen Nielson's advice (from the bug report):

I would recommend that you dump the stack for all running threads when
you experience this problem.  This can help identify what is causing the
problem. By reviewing the stack dump for each thread you can determine
whether the problem is due to Tomcat or your application.

On unix you do a kill -QUIT {tomcat java pid) to cause the thread stack's 
to be dumped to catalina .out.

A Processor for Tomcat runs your application code, delays in your code
can cause additional processor threads to be created to handle new requests.
Possible application or configuration problems which can delay requests:

   Connection delays due to networked services such as a db.
   Connection delays due to running out of pooled resources.
   Thread synchronization deadlocks.
   A cascading affect where many new processors get created due to
   excessively long JVM Garbage Collections.  start java with
   -verbose:gc to detect this.

Ted explains how to plug JRocket into an existing JDK 1.4.1 JRE:

Ted Neward: First, I created a subdirectory in my standard JDK 1.4.1 install structure, j2sdk1.4.1/jre/bin/jrockit. Into this directory I copied the jvm.dll and Xusage.txt files that came from JRockit itself. Then, after adding a line "-jrockit KNOWN" into my JDK 1.4.1's jvm.cfg file (so that the launcher would recognize "-jrockit" as a viable command-line option), I fired up the standard Sun JDK java launcher with the command line "java -jrockit Hello". Sure enough, it worked.

Cameron Purdy commented on FreeRoller performance recently:

Apparently, over half the load is on the RSS/XML side (not the "interactive web/HTML" side,) and it's enough to completely saturate their T1 connection. Wow! And it's running on an eMachine (a US$200 "Walmart computer") that was given up after it had served its useful life. So in a way, it's amazing that it runs at all. However, Anthony has plans to cluster it, and with a couple of commodity servers to run on (plus enough bandwidth?), it should be back flying again, hopefully before too long.

Recently, I've been helping Anthony Eden keep FreeRoller up and I've been watching the logs and the stats. Cameron is right, FreeRoller gets a hell of a lot of RSS traffic and FreeRoller is running on a pretty light-weight machine: a 450Mhz Pentium II with 256MB RAM.  This might be a fine an Apache/mod_perl app, but for big beefy Roller/Tomcat/Struts/Velocity/Castor it's just not enough.

Still, I'm doing what I can to improve performance. For example: this weekend I added caching to FreeRoller's RSS feed. To do this I had to add: 1) handling for the IF-MODIFIED-SINCE header in the PageCache ServletFilter and 2) a new Filter Mapping to map '/rss/*' to Roller's OSCache based PageCache Servlet Filter. This should speed average RSS response time, reduce memory usage, and database access. It may have some effect on the FreeRoller RSS feeds, so if you notice a problem in a FreeRoller RSS feed let me know about it.

From Computer World's interview with BEA's CEO Alfred Chuang:

What Linux trends are you seeing with BEA software?

Huge adoption curve climbing very fast for BEA over the last six to nine months. A lot of focus in the financial services marketplace, where there's a lot of experimentation and initial deployment going on with Linux on Intel. And I think the motivation in that arena is simplification and cost reduction, so they are looking to buy significantly less expensive hardware.

What's the breakdown of platforms on which BEA software is running?

About 50% is on Sun, and about 23%, 24% is on Hewlett-Packard. Hewlett-Packard has both Intel and non-Intel platforms in there. And then it drops off pretty quick. IBM hardware, I think, is 5% or 7%. In some countries, we sell a lot of IBM's hardware.

What about the Linux operating system?

Linux is around the 15% to 20% range, which has climbed pretty quickly.

Get 'em while they're hot. The Server Side reports that Sun has finally released <a href= "http://jcp.org/aboutJava/communityprocess/final/jsr031/index.html">JAXB 1.0. The JAXB reference implementation <a href= "http://java.sun.com/webservices/docs/1.1/ReleaseNotes.html#redistribute">is redistributable, but is it open source?

The Server Side also reports that Sun has released Java Server Faces (JSF) Early Access 3, a tutorial, and a public draft of the JSF spec.

Frank Boosman: But there is another Silicon Valley rising from the ashes -- a new generation of companies starting and growing up, born out of bad times. These companies are the seedlings poking through the ashes of a devastating fire. When the fire is a memory, they'll be the tallest trees in the new forest.

As usual, Matt is on top of the Jakarta news. He notes that new Tomcat 5.0.1 alpha and Ant 1.5.2 are both available for download.

<a href= "http://www.computerworld.com/developmenttopics/development/java/story/0,10801,78943,00.html">Computer World summarizes the Meta Group study by writing that Borland, IBM, and Oracle are the "clear leaders" while Sun and Jetbrains are the "distant challengers."

I developed Roller without unit testing, but I'm not going proceed with the Hibernate implementation without having tests in place. Last night, I checked in tests for the Roller NewsfeedManager, the simplest of the Roller backend managers. The next step is to write or generate Hibernate mappings for the NewsfeedData object and code up a new NewsfeedManagerImpl.

While I was working on that, Matt implemented remember me, added email notification for comments, and upgraded his site to the latest Roller from CVS (not even I am ready to do that ;-)

Finally, I'm ready to start working on the Hibernate implementation of the Roller Weblogger backend. I ended up doing a lot more refactoring than I had intended. I told you about how I moved code from the Castor implementation into abstract base classes that can be used by the Hibernate implementation. Now I'll describe the changes that I made in the Roller build and code-generation process.

In the original Roller build process, illustrated in Figure 4 of the article Building a J2EE Weblogger, I used abstract javax.ejb.EntityBean classes as the meta-data basis for generating code via the XDoclet EJBDoclet task. I was subverting EJBDoclet: using it to generate Data Objects, Struts Forms, and Castor mappings but not using any of it's EJB output.

That worked pretty well, but eventually it became a problem. The generated Data Objects were just dumb data-holders and, over time, we realized that they need to be smarter "business objects." The Data Objects were regenerated on every build and that made it diffucult, if not impossible, to add new methods, new logic, and new collections.

The new Roller build/code-gen process

<img src="http://www.rollerweblogger.org/resources/roller/xdoclet-roller-sm.jpg" alt="Diagram of Roller build process">

The new Roller build process, or at least the code-generation part of it, is shown above. We now start with some hand written "Plain Old Java objects" or POJOs. We still have to subvert EJBDoclet because the XDoclet <strutsform> and <castormapping> can only exist inside EJBDoclet.

I had to use Matt Raible's patched version of <strutsform> (from his struts-resume example) because the one in XDoclet 1.2b2 works only if the source class extends javax.ejb.EntityBean and the new Roller POJO's don't do that. Matt and I consider this to be a bug in EJBDoclet, but I'm not sure the XDoclet guys agree. Maybe Roller should define it's own <strutsform> that works on any POJO and that inserts validator tags (something Matt also added in his struts-resume example).

The Roller build/code-gen process is still not perfect, but it's "good enough" for me to begin my Hibernate implementation of the Roller Business Tier interfaces. I'll be blogging as I go so stay tuned.