Blogging Roller

Latest Links - misc

2008-06-17T19:00:30+00:00

It's time to catch up on blogging and I'm going to start by going through my backlog of links and adding some commentary, but not in this post; these are miscellaneous links that don't fit nicely into my other posts.

Slashdot | Why Google Should Embrace OpenOffice.org
"If Google really wanted to deliver a knockout punch to Microsoft, it would integrate OpenOffice with Google Docs" -- and pump some money into OOo development
Headius: The Power of the JVM
"And while JRuby and Groovy will probably spend the next few months one-upping each other, we've both proven something far more important: the JVM is an *excellent* platform for dynamic languages. Don't let anyone tell you it's not."
SpringSource Blog - Open Source, Open Strategy
My 141 char summary: open source good, but we need money. we won't relicense existing parts, but we'll GPL the full-stack so we alone can ride it to the bank
Kohsuke Kawaguchi: GlassFish v3 just got embeddable
"GlassFish glassfish = new GlassFish(); glassfish.minimallyConfigure(8080);"
Derby: Using the bulk import and export procedures
closest thing to mysqldump in Derby land
Core team statement on replication in PostgreSQL
"it is time to include a simple, reliable basic replication feature in the core system"
Distributed Caching with Memcached
Nice how-it-works article on memcached.

HOWTO: Configure Roller to use Memcached

2008-03-06T17:43:19+00:00

In my last post, I explained the details of Roller cache configuration and I mentioned that Roller's caching system is pluggable but I didn't explain what that really buys you. Basically, what it means is that you can replace Roller's in-memory LRU caching mechanism with something else. In this blog entry, I'll explain how to do that and specifically, how to replace Roller's built-in cache with the popular Memcached caching system via the Roller Memcached plugin, which, like the rest of the Roller caching system was written by Allen Gilliland for use on blogs.sun.com.

First, what is Memcached? Here's what the Memcached web site says:

: "Memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

: Danga Interactive developed memcached to enhance the speed of LiveJournal.com, a site which was already doing 20 million+ dynamic page views per day for 1 million users with a bunch of webservers and a bunch of database servers. memcached dropped the database load to almost nothing, yielding faster page load times for users, better resource utilization, and faster access to the databases on a memcache miss."

Lots of big sites use Memcached for distributed caching. In addition to LiveJournal, there's also Wikipedia, Slashdot, SourceForge and our very own blogs.sun,com. If you're wanting to use Memcached, then you're in good company.

Why would you want to use Memcached with Roller? Well, if you are running a large Roller site and you've got multiple instances of Roller running then you might run into problems with cache consistency. A blog update that hits server A will update the cache on server A, but server B will still have the old blog data for some time. Using a distributed cache will solve this problem. Another, hopefully much less common, reason for wanting to use Memcache is to avoid using the JVM heap -- some folks have experienced some weird problems with the JVM heap management and might want to use plain old C memory management instead via Memcached.

Download and install Memcached

The first thing you need to do is to get yourself a copy of Memcached. You can get downloads and docs on the Memcached web site. You can build it yourself if you're a manly man or a womanly woman, but there are some much easier options.

For example, on Ubuntu or Debian, you can install Memcached with one simple command:


   apt-get install memcached

Or, if you're on Solaris and you're a Blastwave user, then you can do this:


   pkg-get install memcached

On Mac, you can probably do something similar with Fink or MacPorts. There's also a Win32 version for those less fortunate.

And, of course, you'll have to install Memcached on each machine on which you want to act as a cache server.

Setup your Memcached caches

Now that you've got Memcached installed, you need to start it up.

For example, let's say you want to have two 2GB caches running on your two machines with IP addresses 10.0.0.40 and 10.0.0.41. One one maching you'd run this command to start the Memcached daemon with 2048M RAM and listening on port 11211.


$ ./memcached -d -m 2048 -l 10.0.0.40 -p 11211

And on your other machine 10.0.0.41, you'd start Memcached with this command:


$ ./memcached -d -m 2048 -l 10.0.0.41 -p 11211

And, you'll almost certainly want to set things up so that your Memcached daemons start up whenever your servers start up. On Solaris or Linux that means adding things under the /etc/init.d directory. You'll have to figure that out for yourself.

Download and install the Roller Memcached plugin

Next, you need to get yourself a copy of the Roller Memcached plugin and install it into Roller. You can get it from the Roller Support project. Go to the project's file download page and navigate to Roller Support 4.0 / Plugins, or just use the link below:

https://roller.dev.java.net/.../roller-memcached-4.0.tar.gz

Unzip that file and add the two jars within to your Roller install's WEB-INF/lib directory. Assuming that the Roller context is in the directory $ROLLER_HOME, you might do that like so:


$ tar xzvf roller-memcached-4.0.tar.gz 
$ cp roller-memcached/* $ROLLER_HOME/WEB-INF/lib

Configure the Memcached Plugin and Roller caching

The final step is to configure Roller to use Memcached for caching. To do that, you simply add some properties to your roller-custom.properties override file. To use Memcached for all of your caches, set the following property:

   

   cache.defaultFactory=\ 
      net.java.roller.tools.cache.memcached.MemcachedLRUCacheFactoryImpl

If you don't want to use Memcached for all of your caches, then configure it for just the caches you want. Refer to my previous post about Roller caching for details, but basically you just set the factory for the cache to use the Memcached cache factory. For example, to use Memcached for caching feeds only, you'd add this instead of the above:


   cache.weblogfeed.factory=\ 
      net.java.roller.tools.cache.memcached.MemcachedLRUCacheFactoryImpl

You also need to tell the Roller Memcached plugin how to find the Memcached servers. To continue with the example we started above with servers 10.0.0.40 and 10.0.0.41, you could add this to set the default Memcached servers for all caches:


cache.memcached.default.servers=10.0.0.40:11211, 10.0.0.41:11211

You can also set the servers to be used for specific caches . For example, here we set the weblog feed cache to use two different Memcached servers from that used by the weblog page cache:


cache.memcached.weblogpage.servers=10.0.0.40:11211, 10.0.0.41:11211 
cache.memcached.weblogfeed.servers=10.0.0.50:11211, 10.0.0.51:11211

Fire it up!

Now you're ready to start up Roller. You might want to add some debug logging the first time around just to make sure things are working. Roller uses Log4j caching and you can add the Log4j properties directly to your roller-custom.properties file. For example, to enable debug caching for the Roller Memcached plugin you'd add this:


log4j.category.net.java.roller.tools.cache.memcached=DEBUG

And that's that. Let me know via comments if I've left something out or gotten something wrong. I'd also like to hear if you've had some success with Roller and Memcached.

HOWTO: Configure caching in Apache Roller

2008-03-03T12:15:00+00:00

Since the early days, Roller has included a pluggable caching system for blog pages and feeds. In Roller 2.1 (early 2006), Sun's Allen Gilliland rewrote the whole cache system and made it much more flexible and much easier to configure. But, apart from comments in the configuration file, we never provided any documentation for the cache system. In this post, I'll start to correct that. I'll explain the basics of how the cache works and how to configure it.

Overview

Displaying a blog page can take dozens of database queries and database queries can be expensive. They take time, consume CPU cycles and typically use network bandwidth. Roller's built-in caching system addresses this problem by caching generated pages and feeds. By default, Roller caches pages and feeds in memory using a Least Recently Used (LRU) algorithm and by default caches are configured appropriately for a 100 blog system. If you are running a site with more blogs or a very high-traffic site, you should consider changing the caching configuration. First, let's discuss how the caches work.

Cache invalidation and expiration

When Roller generates a page, it puts a copy of that page in a cache. The next time that a request comes in for that page, Roller returns the page from the cache. When a blog changes, Roller invalidates the blog's cache enties, i.e. it throws that blog's pages out of the cache. And by default, when the cache is full and we need to add a new entry to the cache, we push out the least recently used entry in the cache to make room; that's the LRU algorithm I mentioned before.

Sometimes, a blog page includes things that change frequently like a list of referrers or a server-side hit counter or data from some other source. We don't want to invalidate a blog's cache entries every time a hit is counted. That would defeat the purpose of the cache. So, by default Roller uses an expiring cache that automatically invalidates cache entries after timeout period.

Cache configuration

To configure the Roller caches, you add properties to your roller-custom.properties properties override file. You can learn more about this override in Section 6 of the Roller 4.0 Installation Guide and you can find a complete list of the properties you can override in Section 11.

First, let's cover the default caching mechanism. If you're running a large and high-traffic site, you might want to consider using the non-expiring cache or setting the cache timeout very high (4, 6 or 12 hours). Here's how you tell all caches to use the non-expiring cache:


   cache.defaultFactory=org.apache.roller.util.cache.LRUCacheFactoryImpl

However, if you do that, then blogs that use Roller's built-in hit counter or that display referrers will not be updated as often as your users would like. So, you might want to consider removing the #showReferrersList() macro from any themes in use on your site.

Configuring Roller's four page and feed caches

You can configure caching differently for the different types of pages and feeds produced by Roller. There are four separately configurable caches. Here are their names and an explanation of each:

weblogpages: this cache is used to cache weblog pages
weblogfeeds: this one is for weblog RSS and Atom feeds
sitewide: this is for the aggregated front-page blog and it's RSS/Atom feeds
planet: this is for feeds produced by Roller's built-in aggregator

And for each one of these caches you can configure these properties:

enabled: for debugging purposes you can completely disable a cache by setting it's enabled property to 'false'
size: sets the total number of entries allowed in a cache, each entry holds one page or feed response.
timeout: the number of seconds that an entry is allowed to remain in the cache. After this time expires the entry will be removed from the cache.
factory: set the classname of the cache factory to be used for this cache, otherwise default cache will be used

Cache property names follow the pattern cache... The best way to understand how this works is to look at the default cache configuration used by Roller:


# Weblog page cache (all the weblog content)
cache.weblogpage.enabled=true
cache.weblogpage.size=400
cache.weblogpage.timeout=3600

# Feed cache (xml feeds like rss, atom, etc)
cache.weblogfeed.enabled=true
cache.weblogfeed.size=200
cache.weblogfeed.timeout=3600

# Site-wide cache (all content for site-wide frontpage weblog)
cache.sitewide.enabled=true
cache.sitewide.size=50
cache.sitewide.timeout=1800

# Planet cache (planet feeds)
cache.planet.enabled=true
cache.planet.size=10
cache.planet.timeout=1800

The default cache configurations above are setup for a 100 weblog system. To some extent, this is guess-work. For example, we've decided to cache 4 pages and 2 feeds for each blog. That's how we arrived a cache.weblogpage.size=400 and cache.weblogfeed.size=200. And we've decided to cache blog entries for 30 minutes and feeds for one hour. That's how we arrived at cache.weblogpage.size=400 and cache.weblogfeed.timeout=3600.

You might decide to do things a little differently on your Roller system. Copy the properties above to your roller-custom.properties file and set them to values you thing are appropriate for number of weblogs, average page size, traffic levels and JVM heap size of your Roller installation.

Conclusion

Roller default cache configuration will work well without modification for a small to medium size Roller installation, but for large high-traffic sites you should increase cache sizes and think carefully about timeouts. And if you're running Roller in a cluster you might want to consider using a distributed caching system like memcached. I'll discuss that in my next HOWTO.