Blogging Roller

Continuing on the childish theme, my 6-year old son Alex claims to have invented and popularized this silly chant at his school: "kids are the best, we know true, put us all together and we shout BOO!"

That's right, Java/J2EE is a minivan. It is not sexy or hip like Ruby or Python. It is sensible, practical, gets the job done, and is relatively easy on the software industry environment. C#/Dot-Net, of course, is the SUV: crass, macho, and a real environmental hazard, especially to endangered species soon to be Redmond roadkill like Borland, Ximian, SourceGear, etc.

A number of people have asked how Freeroller suddenly became so fast and so stable. The answer is Roller 0.9.8. Freeroller had been running Roller 0.9.7, which had a history of performance and stability problems from the start.

The first problem we experienced with Roller 0.9.7 wasn't noticed by the users, but our gracious hosts at The Javalobby noticed an unusually high load on their server coming from the Roller and MYSQL processes. Later, as the number of bloggers blogging and readers visiting the site grew, users started to notice unpredictable behavior. Spurious 404 errors, sluggish performance, and frequent downtime started to irritate everybody.

Roller 0.9.8 seems to have resolved all of these problems. Let's take a look at the changes made since Roller the initial 0.9.7 release starting with the changes that were applied to Freeroller before the Roller 0.9.8 upgrade:

• Caching weblog update times. Because Roller uses page caching, most content is served out of the page cache and for most requests there is almost no need to hit the database. For most requests we only hit the database to bump up hit counters and to determine last-update-time for RSS newsfeeds. Still, Roller was killing MYSQL and the load from Roller and MYSQL was dragging the Javalobby server down. Since RSS newsfeeds account for more than 60% of all traffic, I decided to cache those last-update-times. Javalobby noticed an improvement in load, but load was still a little high.

• The Open session in view pattern. With new users signing up every day and a number of more high-profile bloggers starting to draw in the hits, Freeroller really started to drag and users started to notice. The Roller development team started to wonder "what have we done wrong?" We eventually concluded that Roller uses too many persistence sessions, which roughly correspond to database connections, per request. Lance implemented the open session in view pattern in the Castor and the Hibernate persistence implementations in the Roller main branch. I backported the changes to the Roller 0.9.7 branch and deployed them to Freeroller only to find a minimal performance increase.

• OSCache 2.0-beta upgrade. Around that time, the OSCache team relased a beta of OSCache 2.0 and Hani told me that the old OSCache was rubbish and had a terrible memory leak, so I switched Roller and Freeroller over to the new OSCache, configured it for memory caching plus unlimited disk cache, and noticed a pretty big leap in performance and decrease in system load. Still, the stability was poor and spurious 404s continued.

Those changes were important changes, but they did not fix the performance and stability problems. Now, let's move on to the changes that were added by the Roller 0.9.8 upgrade:

• Numerous small fixes. While I was fussing with Freeroller and Roller 0.9.7, Lance and Min were looking for further performance improvements in the Roller 0.9.8 branch. Min fixed the Roller calendar control to keep it from hammering the database as it use to. They both worked to change the persistence layer interfaces to return lists instead of arrays, since both Castor and Hibernate return lists, list to array conversions are just silly in this case.

• Database indices. At some point during Roller 0.9.8 development somebody (Jim Smart?) contributed a patch for the Roller database creation script and added indices where indices were missing. I didn't think much of it at the time, but this was a pretty significant change.

• Hibernate/JCS caching. Freeroller was getting worse and worse, even causing Freeroller founder Anthony Eden to leave and, believe it or not, even causing the Hibernate team to feel the heat. Some were suggestingthat perhaps Hibernate did not scale. Gavin King stopped by the Roller dev-list and suggested that we try using proxies and JCS caching. I didn't get far with proxies, but it was easy to turn on JCS caching. Unfortunately, JCS caching by itself didn't seem to help much.

So, what change made the big difference in performance? It can't have been only update-time caching, open session in view, and OSCache 2.0 changes because they were all in place before the Roller 0.9.8 upgrade. It can't have been Hiberate/JCS caching because stress testing showed that JCS actually slowed down the system a bit. All of these fixes played a part in the performance improvements, but because of the greatly reduced load on MYSQL I think the database indices made the biggest difference.

The Roller vs. WebloggerZilla fiasco ended peacefully.

This new Roller release provides a dramatic performance increase due to addition of database indices, proper use of persistence sessions, and a new version of OSCache. New features include Weblog.com ping and integrated Lucene weblog search. Thanks to all that helped out with the release. For details, see the change notes. You can download the release from the project's SourceForge download page.

I can't believe that people feel the need to rag on Gavin King, the lead developer of Hibernate, because he joined the JBoss Group. They say he "sold out" and joined the dark side. What a crock. Do we really have to break up into JBoss, CDN, Open Symphony, and Jakarta camps and sling mud at each other? No, we don't! And really, who wouldn't quit their day job when they finally find a way to make their night job pay? There is no reason be upset, people. Unless, of course, they start charging for Hibernate docs. Then, my friends, heads will roll.

has gone too far, matey.

The FreeRoller performance and stability problems seem to be behind us, so I'll be making a the Roller 0.9.8 release this weekend. A number of people have asked how we achieved the dramatic performance increase on FreeRoller, so I'll try to summarize the performance fixes that were made between 0.9.7 and 0.9.8 in a later post.

The lights have been flickering and the cable modem has been dropping in and out, but I continue to work and the kids continue to enjoy the first of two days off. The wind and rain are supposed to be the heaviest from 3PM to 6PM here in Raleigh. From where I sit, I can see a big stand of tall oak and pine trees whipping around in the wind. I'm hoping that things don't get any more exciting than this.

is the word of the day in more ways than one. Confluence is a new web-based knowledge management and group collaboration application apparently built upon concepts from on content management, weblogging systems, and wikis. Developed by Atlassian, the company that developed the popular Java-based JIRA bug tracker and java.blogs, Confluence is currently in alpha testing.

Today was cool and clear fall day here in North Carolina. No sign of the approaching storm. I live in Raleigh which is a couple of hundred miles from the shore, but the storm could track close enough to us give us a dose of 60 mile-per-hour winds, power outages, and downed trees. I'm hoping that the two giant oak trees in our yard will survive and will not land on our house. My hurricane preparations? I stowed all of our lawn furniture, bought extra charcoal for the grill, and bumped up the maxProcessors thread-count on FreeRoller from 75 to 200 threads.

I walked out to my mailbox today to find that FreeRoller has vaulted everybody's favorite blowhard, the Howard Stern of Java blogs, bile-blogger Hani Suleiman to worldwide fame in the pages of Java Developers Journal. Hani's editorial, is just another in a series of unnecessary open-source-ain't-all-it's-cracked-up-to-be pieces from JDJ. The motivation for this most recent open source backlash? I guess JDJ feels the need to please it's advertisers and Hani feels the need to poke a little stick in the eyes of the JBoss group and Andy Oliver.

I've been googling around for some guidelines for setting the number of request-processing threads on a Tomcat site. So far, I've found nothing. FreeRoller had been set for 75 threads, but it seems to run out at that level. Does this by itself indicate a problem or is 75 threads too low for a relatively high volume blog site like FreeRoller?

I just realized that I am sitting in a chair that was given to me by the now CTO of Amazon.com. He also gave me a computer and a "pink slip." I'm not bitter. It is a quite comfy chair.

Hitting the racks this fall...

<a href= "http://www.wiley.com/WileyCDA/WileyTitle/productCd-0471463620.html"> <img class="photo" border="0" align="right" src="http://www.rollerweblogger.org/resources/roller/book-josp.png" alt="Book cover: Java Open Source Programming" /> Java Open Source Java Programming by Joseph Walnes, Ara Abrahamian, Mike Cannon-Brookes, and Patrick A. Lightbody

I had the priviledge of reviewing a number of chapters from this book and found it to be not only a good introduction to WebWork, Sitemesh, and Hibernate, but also a great primer on test driven development and the use of mock objects in unit testing. It takes real disipline to do true test-first development and the authors of this book certainly have the disipline. The authors do whatever is necessary, sometimes writing scads of throw-away code along the way, to ensure that a test-first safety net is in place for each and every new development.

<img class="photo" border="0" align="right" src="http://www.rollerweblogger.org/resources/roller/book-projsp.png" alt="Book cover: Pro JSP" /> Pro JSP By Simon Brown, Sam Dalton, Daniel Jepp, Dave Johnson, Sing Li, and Matt Raible.

And now, from the shameless plug department... While you are book shopping you should also pick up a copy of Pro JSP, covering JSP 2.0, JSTL, database access options, Struts, XDoclet, and more. If you do, I'm sure I won't regret it.

O'Reilly's new <a href= "http://j2eegeek.blogspot.com/2003_09_01_j2eegeek_archive.html#106350631823650194">Tomcat book is a must have. He also has some nice screenshots from of the BEA JRocket Java VM management console and runtime analyzer. I would try JRockit, but I hate that whole -- it's a free download but you have to give us your email address and purchase is required for real use and we won't tell you the price unless you talk to a sales person on the phone -- deal.

Sam Ruby provides an excellent overview of RSS and covers it's historical quirks and current usage in his RSS in depth slides from the Seybold conference.

We spent the first week of September 2001 at Ocracoke, one of the North Carolina barrier islands. Ocracoke is a three hour ferry-boat ride from the coast and the town of Ocracoke is tiny, so it feels so remote, disconnected, and isolated from the problems of the rest of the world. At the crack of dawn each morning, I would go out for a run. When I returned, I would take one or both of the boys out for a walk down the road, past the cute little light house, past the pelicans, and down to the little harbour, known as Silver Lake. One morning we found a bunch of kitties in the woods by the road, one morning we walked to the other side of town for breakfast, and every morning we saw a beautiful sunrise. This is what I would prefer to remember about September 2001.

Alex and Linus, on one of those morning walks.

<a href= "http://blogs.atlassian.com/rebelutionary/archives/000287.html#comments"> Rebelutionary found this one. Ruckyou.com has taken Roller added a bunch of cool new themes (and who knows what else) and made an awesome looking Rugby community site.

A half-dozen very professional looking new Roller blogs have appeared at la Repubblica.it: Estremo occidente, Pulp Fiction, Rimbalzi, Blog Trotter, and Media-Trek. Now, somebody please tell monolingual me: what is la Repubblica.it?