Blogging Roller

I talked to Rick Ross earlier today and learned that JRoller is down due to hardware problems. Rick indicated that the system might be back up later tonight. Update: JRoller is back.

After a little struggle my site is running the absolute latest Roller code. Unfortunately, SourceForge CVS is down so I cannot commit the half-dozen or so fixes I made in the process of my upgrade and SourceForge mailing-lists are down so I can't announce this wonderful new achievement to the folks who actually care.

BTW, my site was halfway between Roller 0.9.8 and Roller 0.9.9 so the upgrade was tricky. I haven't tested a Roller 0.9.8 to Roller 0.9.9 upgrade yet, but I'm confident that I'll be able to make that upgrade just as smooth as silk.

Turbulence ahead.... Update: As of Friday morning, I'm still working on a Bookmark/OPML import/export problem.

Is now enabled: http://www.rollerweblogger.org/wiki/rss.rdf

As Lance notes, the Roller code in the Roller CVS main branch, also known as Roller 0.9.9-dev, is in a bad state. The code passes all unit tests, but I'm still working on data model changes. As always, there will be an upgrade script to upgrade the schema from the last release (Roller 0.9.8) to the next release (Roller 0.9.9), but the script might have problems if your database is half-way between 0.9.8 and 0.9.9. Ah, the joys of database-driven webapp development! Anyhow, if you want to try Roller 0.9.9-dev, I would recommend that you wait until you get the all-clear from me (possibly this weekend). Once I wrap up hierarchical categories, I'm going start working on bug fixes for a Roller 0.9.9 release.

BTW, Lance mentioned that I'm working on hierarchical weblog categories. Unfortunately, for a database-based weblogging system like Roller, this is a fairly complex undertaking. In file-system based weblog system, categories are almost trivial. Weblog categories are just directories sitting in the file-system. The hierarchy comes for free. Want a post to exist in multiple categories? Just create a symbolic link. For Roller, the hierarchy must be modeled as associations between tables. Without some trickery, retrieving all weblog entries in a category and all subcategories requires recursive descent with a database query at every level. The trickery is now in place and abstracted in such a way that I can use it for all hierarchies: weblog categories, bookmark folders, and maybe someday, page templates.

These days it is pretty unusual to find JRoller down for more than a minute or so, but down JRoller is. JavaLobby is investigating the problem, which seems to be an ISP problem rather than a Roller software problem. Update: JRoller is back up, but haven't heard the cause of the outage.

Rick Ross and I were trying to diagnose some heavier-than-usual load on JRoller last night by watching the MySQL query log. We noticed that some queries were still examining way too many rows and based on this we came up with a couple of additional indexes. After we applied the new indexes, load returned to normal levels (i.e. less than 0.1). These should work on either Roller 0.9.8 or Roller 0.9.9-dev and they will be included in future releases in either branch:

create index userrole_username_index on userrole( username );
create index weblogentry_pubtime_index on weblogentry( pubtime, publishentry, websiteid );

David Czarnecki responds in detail to some reader comments in my October 2003 Blojsom vs. Roller post. Things have changed a bit in the months since that post. Roller and Blojsom are becoming more alike in terms of features, but the architectures are still very different. Roller is classic database-driven J2EE webapp with the patterns and the frameworks and the associated warts. Blojsom plays off of the extensible and file-system based Bloxsom architecture. Both of these architectures have strengths and weaknesses.

Lance, Matt, and I have agreed to serve as judges for the upcoming JRoller Theme Contest. Arjun Ram is coordinating the contest so keep an eye on his blog for further details.

I added some additional logging to Roller's RefererManagerImpl.java to help track down the dayhit rollover bug, stopped JRoller, was called upstairs to deliver nite-nite hugs and kisses to the kidlets, returned downstairs to find that Andi had queued up the Sopranos DVD, and completely forgot about JRoller for about an hour or so. Rick and Matt restarted the server for me. I know, that sounds completely unprofessional, but this is life. My most sincere apologies to all who were inconvenienced.

Roller has changed quite a bit since the Roller article was published on the O'Reilly site. Roller has moved from a two layer architecture with a Presentation Layer and a Business Layer to a three layer architecture with Presentation, Business, and Persistence Layers. This article covers the transition and will bring you up to date with the current Roller persistence architecture.

Then, as now, the Roller Business layer was a small collection of interfaces. The Roller interface was the entry point and provided access to the various Roller manager interfaces. Back then, the Business Layer interfaces were implemented in the org.roller.business package and used the Castor O/R mapping framework APIs directly. The diagram below illustrates this point. The org.roller.business package implements the Business Layer intefaces defined in org.roller.model and depends on org.exolab.castor.

<img src="http://www.rollerweblogger.org/resources/roller/roller-2layer.png" alt="Roller's original 2-layer architecture" />

Earlier this year, during Roller 0.9.7 development, we decided to create an implementation of the Roller Business Layer using the Hibernate O/R framework. We moved the Castor implementation into the org.roller.business.castor package and added the new Hibernate implementation in org.roller.business.hibernate. In the process we noticed a lot of code duplication and we tried to move common code back up into org.roller.business, but we had limited success. The diagram below illustrates this situation: separate Castor and Hibernate implementations of the Roller Business Layer with some shared code.

<img src="http://www.rollerweblogger.org/resources/roller/roller-2impls.png" alt="Two implementations of the Business Layer interfaces" />

Since Roller 0.9.7, the Roller Business Layer implementation has been evolving and moving towards that three layer architecture mentioned at the start of this article. We introduced a PersistenceStrategy interface and a persistence package in org.roller.persistence. Over time we have slowly moved almost all Castor and Hibernate specific code behind the PersistenceStrategy interface. I say almost because, there are a couple of cases where the Persistence Layer abstraction is not powerful enough to handle the queries needed by the Business Layer. Specifically, the RefererManager.getHits() and RefererManager.getDaysPopularWebsites() methods need to drop down to raw SQL because, apparently, neither Castor nor Hibernate can handle the queries we need to do.

The diagram below illustrates the current situation. The Business Layer calls upon the Persistence Layer for object storage, retrieval, and queries. The org.roller.business.castor and org.roller.business.hibernate packages are shown in grey because they still exist but are on the way out.

<img src="http://www.rollerweblogger.org/resources/roller/roller-3layer.png" alt="Two implementations of the Business Layer interfaces" />

Thus far, we've been looking at things at the package level. Let's zoom in on the org.roller.persistence package and take a close look at the key interfaces in that package: <a href= "http://www.rollerweblogger.org/javadoc/org/roller/persistence/PersistenceStrategy.html"> PersistenceStrategy, <a href= "http://www.rollerweblogger.org/javadoc/org/roller/persistence/QueryFactory.html"> QueryFactory, and <a href= "http://www.rollerweblogger.org/javadoc/org/roller/persistence/Query.html"> Query.

<img src="http://www.rollerweblogger.org/resources/roller/persistence-api.png" alt="Roller persistence and query API interfaces" />

PersistenceStrategy supports storage, retrieval, and removal of objects. To execute a query, you create a Query object using the QueryFactory, create a where clause by adding Conditions together, set a limit if desired, set an order-by if desired, and call the Query.execute() method. At that point, the Query implementation will generate an HQL query for Hibernate or an OQL query string for Castor and will call the appropriate method to execute the query. Below is some example code to illustrate typical usage of the Query API.

    public List getTodaysReferers(WebsiteData website) throws RollerException {
        if (website==null ) throw new RollerException("Website is null");
        QueryFactory factory = mStrategy.getQueryFactory();
        Query query = factory.createQuery(RefererData.class.getName());                
        Condition specifiedUser = factory.createCondition(
            "website", Query.EQ, website);            
        Condition hasDayHits = factory.createCondition(
            "dayHits", Query.GT, new Integer(0));            
        Condition userEnabled = factory.createCondition(
            "website.user.userEnabled", Query.EQ, new Boolean(true));            
        List conditions = new LinkedList();
        conditions.add(specifiedUser);
        conditions.add(hasDayHits);
        conditions.add(userEnabled);                
        query.setWhere(factory.createCondition(Query.AND, conditions));            
        query.setOrderBy(Query.DESC, "dayHits");       
        return query.execute();
    }

The PersistenceStrategy interface is a fairly simple abstraction, sort of a least common denominator approach, but it is expressive enough to handle all of the requirements of Roller. This approach allows the Roller Business Layer to be completely independent of the underlying persistence mechanism. This is really nothing new. PersistenceStrategy is essentially a generic Data Access Object (DAO). I use the word "generic" because most DAOs I've seen are specific to one type of object, a CustomerDAO or EmployeeDAO for example, but PersistenceStrategy handles any persistent object in the system. I'm interested to see how this approach holds up. Will we eventually find the simple API to be too limiting? Will it be possible to implement PersistentStrategy with other persistence technologies such as JDO?

As part of Roller 0.9.9's new boomkark management features, I have also added the ability to export the bookmark folder hierarchy as OPML. I implemented this by changing Roller's RSSServlet into a FlavorServlet that serves "flavors" defined by Velocity templates found in the classpath at under /flavors and by adding a new flavor called opml.vm.

For example, the following URL will retrieve all of my folders in OPML format:

• /flavor/roller?flavor=opml

The following URL will retrieve only the contents of the my "/Subscriptions/Blogs - blogroll" folder:

• <a href= "http://rollerweblogger.org/flavor/roller?flavor=opml&path=/Subscriptions/Blogs%20-%20blogroll"> /flavor/roller?flavor=opml&path=/Subscriptions/Blogs%20-%20blogroll

And finally, the following URL will return my RSS2 feed:

• /flavor/roller?flavor=rss

Note that the FlavorServlet defaults to RSS and is mapped to /rss as well as /flavor so all existing Roller RSS URLs will continue to work.

• /rss/roller <-- still works

There is also a new macro #showOpmlLink that renders an OPML icon which links to the bookmark folder of your choice. For example, near the bottom of my Bookmarks page there are OPML links, one for my bookmarks and one for my subsriptions. These OPML links are displayed by the following template code:

<p>You can download my entire bookmark collection in OPML format 
using the links below:</p>
#showOpmlLink("/Bookmarks")Bookmarks (shown above)<br />
#showOpmlLink("/Subscriptions")RSS Subscriptions (not shown)

Over the past month or so, I've been re-working Roller's bookmark management features. As part of this work I added a new Query API so that we no longer have to hard-code Castor or Hibernate query language statements into the Roller Business Layer, I modified BookmarkManager to handle folder hierarchies of arbitrary depth, and I modified the bookmark POJOs to use Castor and Hibernate's lazy-loading facilities. I also rewrote the bookmark management JSP pages and used the JSTL <fmt:message> tag to externalize all strings on the page (for I18N).

Below is the new bookmark manager interface. It works fine now, but it still needs a little fit and finish.

The new bookmark management features will be included in Roller 0.9.9. If you want a preview, check out the new Bookmark Management docs I'm working on.

Now that I've got the hierarchy thing down, I'm turning my attention to the WeblogManager where I will add support for multiple categories per weblog entry and for hierarchical categories.

For RSS and OPML generation... OK, seems to be working now. Erik Thauvin donated his XML/HTML encoding methods to Roller. I just added them to Roller 0.9.9-dev which is running this site now I'll apply them to the Roller 0.9.8 branch for JRoller. Thanks Erik! Update: This is the class Erik donated: TextToHTML.java.

I've always thought it would be cool to package Roller, Jetty, and Hsql-db together with a nice installer and make it really, really easy to download and try Roller. All you'd have to do to start Roller would be to run the installer and run a startup script. So, today I was very happy to see that somebody, by the name of Tarka, has documented the Roller installation process for Jetty. Thanks Tarka.

Jetty appears to be very flexible. The Jetty configuration files allow you to instantiate, configure, and call arbitrary Java objects. I wonder if there is a way to configure Jetty to start HSQL in-process, or if I need to write my own launcher to start/stop Jetty and HSQL? Later... hey, maybe that is what JettyPlus does...

The path to Roller 1.0 is coming into view.

According to the three amigos "architecture is the set of significant decisions about the organization of a software system." Here are some of the more significant decisions that have been made in the development of Roller. Some are questionable, some are bad and have been reversed, and some, as longtime followers of Roller internals will note, are new.

Presentation Layer

• Editor UI will be implemented as Struts actions and JSP pages.
  • JSP pages will use Struts tags, JSTL wherever possible, and custom tags as needed.
  • Header and footer includes will be used (rather than Struts Tiles).
  • JSP custom tags are to be preferered over JSP Scriplets.
• HTML Forms will be implemented with Struts form tags.
  • Struts form tags will be generated via XDoclet from business layer POJOs.
  • Each Struts form bean can copyTo a POJO or copyFrom a POJO
  • copyTo POJO methods will only copy primitives (String, Boolean, boolean, Integer, int, etc.).
• Weblog pages and RSS feeds will be implemented via Velocity templates.
  • Users will use Velocity templates to define pages of weblog.
  • Weblog page macros will be implemented as Veloci-Macros.
• Deployment descriptors
  • struts-config.xml will be generated by XDoclet.
  • web.xml and roller.tld will be generated by XDoclet.
• Interface with Business Layer
  • Presentation Layer will depend on Business Layer via interfaces only.
• Performance
  • Weblog pages and RSS feeds will be cached via Servlet Filter version of OSCache.
  • RSS feeds will obey HTTP IF-MODIFIED-SINCE header.
  • Main page will be cached via OSCache custom JSP taga

• Bad decision, reversed: Struts form beans will each wrap a POJO.
• Bad decision, not yet fully reversed: Editor UI will be implemented via large grained custom JSP tags.

Business Layer

• Business Layer will be defined by a set of interfaces.
• Business Layer objects will be implemented as hand-coded POJOs.
• Business Layer will be Presentation Layer and persistence engine independent
  • Business Layer will not depend on presentation APIs (eg. Struts, Servlet API).
  • Business Layer will use a Persistence Strategy interface
  • Persistence strategy will be implemented for both Hibernate and Castor.
  • Hibernate and Castor mapping files will be generated by XDoclet from POJOs.

• Bad decision, reversed: POJOs will be generated by EJBDoclet from abstract EJB bean classes.
• Bad decision, reversed: Each Business Layer method will open, commit or rollback, and close it's own persistence session.

Persistence Layer

• Open Session in View pattern will be used to ensure maximum of one persistence session per request.
  • At the start of each request, persistence session will be opened via Servlet Filter.
  • At the end of each request, persistence session will be rolled-back and closed via Servlet Filter.
  • Persistence Layer actions must exlicitly commit changes.
  • Persistence sessions will be managed via Thread Local Storage (similar to Thread Local Session).
• Load object by ID will return null if object not found.

<a href= "http://www.jroller.com/page/Sandymountster/20031111#the_roller_weblog_gatorjug_topic"> Mike Levin This month we'll learn about Roller, the open-source weblog. Written entirely in Java, Roller uses many design patterns and java components and has a fascinating architecture. Pizza and refreshments will be served. ...Welcome to the Gator JUG in sunny Gainesville, Florida.

I've been working on my own Roller presentation, now Mike has beat me to the punch.

Lance has been working on improving Roller's plugin support. He also committed some fixes to Roller's RSS 0.91 support. I've also been working on refactoring and enhancing Rollers bookmark management. We been making changes, so now I have to decide what to deploy next to JRoller. Should I support JRoller through a nice safe 0.9.8 branch, or should JRoller stay in the main branch which is now considered to be 0.99-dev? The JavaLobby guys said they want to be on the bleeding edge of Roller development, so perhaps 0.99-dev is the way to go.

Rick Ross has posted what he calls "probable items" for the JRoller terms of service (TOS). I think the items he outlined are quite reasonable. Also on JRoller, Lowem has posted a number of good suggestions for improving Roller including better navigation to past weblog entries, better handling of comments, and showing more weblogs and weblog posts on the Roller main page.