Blogging Roller

Roller 6 released

2019-12-31T15:04:17+00:00

This latest release of Roller includes a new UI that uses Twitter Bootstrap 3 and is based on work I started in 2015 and first committed on December 21, 2015 with this commit: 2da6c3c2e28419f68244e0c362c15be96013d5f9. You can find the details on on the Roller project blog. I got lot of help along the way with testing, fixes, dependency upgrades, Java 11 support and more, so thanks to all that helped make this happen.

Powered by Digital Ocean Kubernetes

2019-02-10T22:42:05+00:00

Just a note to say that I've switched this site over to Digital Ocean Kubernetes service, which is in Limited Availability right now.

Digital Ocean's Kubernetes service is just as simple and well designed as the rest of Digital Ocean. I mentioned before that I rolled my own Kubernetes cluster via Ansible and Kubeadm. Now I can delete all those config files and that's a good thing. Plus, the price is right; I can get by with one $10/month node (1 CPU / 2 GB memory) and a $10/month load balancer.

To get this site up and running I had to deploy four things to my cluster. I installed the NGINX Ingress Controller, Cert-Manager for automatic creation of Let's Encrypt TLS certs, PostgreSQL and my custom build of Apache Roller. All of that went pretty smoothly and I didn't run into and problems that I could blame on Digital Ocean.

Roller 6.0.0-SNAPSHOT

2019-01-26T22:13:38+00:00

Upgraded this site to Roller 6.0.0-SNAPSHOT today, which meant an hour of fiddling around with my private Docker registry, then giving up and using the one free private repository offered by DockerHub and then, another hour of futzing around trying to figure out my PostgreSQL JDBC driver doesn't work anymore (I inadvertently upgraded from JDK 1.7 to 1.8) and why I can't seem to upgrade it (Kubernetes caches Docker images unless you set imagePullPolicy to always). In the end, I got it working. This post is written in the yet to be officially release Apache Roller 6.0.0-SNAPSHOT version.

Side note: the new rich-text editor in Roller is now Summernote and it seems quite nice. I need to tweak it a bit because there is currently no way to set the font or add a link unless you switch to raw HTML mode.

Roller's new web UI

2019-01-21T15:59:00+00:00

About three years ago I decided to modernize and improve the Apache Roller web UI by rewriting the JSP pages to use the Struts 2 Bootstrap tags, which use Twitter's Bootstrap v3 components and JavaScipt. I also wanted to replace all the HTML table-based formatting with div's and Bootstrap, do a bunch of other improvements and make Roller's web UI less clunky and annoying.

Converting Roller's eight-five JSP pages was a big task and I did not have much time for it. That's why it took three years. Ironically, the Roller modernization project leaves Roller three years out of date. Still, I think it is a huge improvement over the Roller v5 web UI and I want to get it released in Roller v6. Currently, this work is available as Pull Request #22 and you can find some screenshots there too. Here's one:

Try it with Docker-Compose

I also did some work to make it super-easy to try the Roller v6 snapshot pre-release for yourself, by using Docker Compose. You don't have to fiddle with Tomcat or PostgreSQL. You can find a simple Dockerfile for running Roller v2 snapshot and a docker-compose.yml file linked below. And you can find a Docker image in my DockerHub repo.

If you want to try Roller v6 snapshot, here's what you need to do:

1 - If you don't aleady have it, install Docker

2 - Create a directory on your computer where you want Roller to store it's data.

3 - Save this file docker-compose.yml to that new directory.

4 - Open a shell in that new directory and run:

docker-compose up

5 - Watch the PostgreSQL and Roller startup logs scroll by

6 - When the log scroll slows go to http://localhost:8080 to access Roller and go through the initial setup.

Alternatively, if you want to try Roller the hard way, you can get the regular-style v6 SNAPSHOT release files here roller/roller-6.0/v6.0.0.

Let us know how it goes

I hope you'll give Roller v6 snapshot a try and let the project know how it can be improved for your use. Send feedback to the Roller mailing lists or ttweet at us at @apache_roller.

Powered by Kubernetes

2018-03-13T14:29:55+00:00

Just a quick note to say that I ditched Docker Swarm and now this rarely updated blog is powered by Kubernetes. Total overkill, I know. Like Roller itself, I did it as a learning exercise. I hope to blog more about what I learned by doing this. For now, here's a quick summary of what I've done so far.

Created a cluster

I created a 2-node Kubernetes cluster on Digital Ocean using some hand-crafted Ansible scripts that call apt-get to install and kubeadm to start Kubernetes. I considered using Typhoon to create the cluster, but I really wanted to learn how to install Kubernetes "from scratch".

Ran two Ingress Controllers

To avoid using Digital Ocean's $20/month load balancer I'm running an Nginx Ingress controller on each node, and pinning containers to nodes using labels and nodeSelectors. I had to borrow Nginx Controller setup files from the Typhoon project because I'm still kind of bewildered by Ingresses.

Deployed my containers

Next, I wrote Kubernetes YAML files for deploying my containers: a private Docker Registry, PostgreSQL and my custom Roller image. Getting the private registry working properly was the biggest challenge. I need private because I don't want to make my custom Roller image public. Next, I'll install Jenkins next for CI/CD of my custom Roller build via the Jenkins Kubernetes plugin.

Let me know if there are any aspects of this that you'd like to see covered in a blog entry, or suggestions for running the cluster without two Ingress Controllers. I've already got a post cooking about installing a TLS secured Docker Registry on Kubernetes.

Powered by Postgresql and Docker Swarm

2017-11-07T21:52:40+00:00

It was somewhat painful but due to some problems with MySQL and Docker, and some general uneasiness with MySQL, I switched this site from MySQL v5.7 to PostgreSQL v10. I also switched over to Docker Swarm. Here's the Docker-Compose file that I'm using now to run this site:

version: '3.2'

services:

   postgresql:
      image: "postgres:10.0"
      ports:
         - "5432:5432"
      deploy:
         resources:
           limits:
              memory: 50M
      volumes:
         - type: bind
           source: /var/lib/postgresql/data
           target: /var/lib/postgresql/data
      environment:
        - POSTGRES_USER=roller
        - POSTGRES_DB=rollerdb
        - POSTGRES_PASSWORD_FILE=/run/secrets/pg_passwd
      secrets:
        - source: db_passwd
          target: pg_passwd

   roller:
      image: "rwo:latest"
      ports:
        -  "80:8080"
      depends_on:
        - postgresql
      deploy:
         resources:
           limits:
              memory: 800M
      volumes:
        - type: bind
          source: /var/lib/roller
          target: /var/lib/roller
      environment:
        - DB_HOST=postgresql
        - STORAGE_ROOT=/var/lib/roller
        - JAVA_OPTS="-Xmx700m"

secrets:
  db_passwd:
    file: ./db_passwd.txt

It was a pain, but sometimes pain = gain and I learned a lot. I'm hoping the site will be a bit more stable now.

Modernizing the Roller UI

2016-06-11T18:28:36+00:00

I don't blog very often but I still find time to work on my blog's software: Apache Roller.

Recently, I decided to focus on improving Roller's ancient Struts 2-based user interface (UI). I had considered adding a comprehensive API to Roller and building a new UI based on that API, but wow that is a huge amount of work. Instead, I decided to modernize the Roller UI by using Twitter's Bootstrap components and CSS styles.

So far, I've devoted a couple of weekends to this work and made some pretty good progress. I'm about half-way done. I'm using the Struts2-Bootstrap plugin, adding better client-side form validation with JavaScript and doing my best to improve the overall user experience. You can see an album of the pages I've done so far on Flickr:
Roller UI with Bootstrap.

I would love any contributions, so if you are interested in helping out, please submit Pull Requests against the bootstrap-ui branch in the Apache Roller repo on GitHub.

Fractal fun with HTML5 and TypeScript

2016-01-11T10:00:00+00:00

MbCanvas is a fun project that I did in 2015: a simple Mandelbrot Set viewer written in Typescript and using the HTML5 Canvas. I did the project to learn more about Typescript and the HTML5 Canvas and I must say, Typescript very nice -- so much easier to read and write than plain old JavaScript, at least for me.

Here's an example image from the viewer.

The project is fairly easy to build if you've got Node and NPM installed, or you can play around with it here: mbcanvas - Mandelbrot viewer in TypeScript

Usergrid-Vagrant updated for Usergrid 2

2016-01-02T11:44:45+00:00

If you're interested in trying the not-yet-released Apache Usergrid 2 you might want to checkout my Usergrid-Vagrant project on GitHub. I just updated the project to support Usergrid 2, using the latest code from the Usergrid "release" brach. The big changes were switching to OpenJDK 8 and adding ElasticSearch. I also rewrote the scripts to use plain old Bash instead of Groovy.

https://github.com/snoopdave/usergrid-vagrant

If you want the old Usergrid 1 Vagrant-file then checkout the "1.x" branch.

Apache Shiro for authentication in Roller

2015-02-09T07:27:50+00:00

This is the third of my 2014 side projects that I'm sharing and one that involves the Apache Roller blog server and the Apache Shiro security framework. You might find this interesting if you're considering using Shiro for authentication and authorization, or if your interested in how security works in Apache Roller.

Inspired by my work with Ember.js in Fall 2014, I started thinking about what it would take to build an Ember.js-based editor/admin interface for Apache Roller. To do that, I'd need to add a comprehensive REST API to Roller, and I'd need a way to implement secrity for the new API. I've enjoyed working with Apache Shiro, so I decided that a good first step would be to figure out how to use Apache Shiro in Roller for Roller's existing web interface.

Working over the winter break I was able to replace Roller's existing Spring security implementation with Shiro and remove all Spring dependencies from my Rollarcus fork of Roller. Below I'll describe what I had to do get Shiro working for Form-base Authentication in Roller.

Creating a Shiro Authorizing Realm

The first step in hooking Shiro into Roller is to implement a Shiro interface called ShiroAuthorizingRealm. This interface enables Shiro to do username and password checks for users when they attempt to login, and to get the user's roles.

Below is the first part of the class, which includes the doGetAuthenticationInfo() method, which returns the AuthenticationInfo for a user specified by an AuthenticationToken that includes the user's username. In other words, this method allows Shiro to look-up a user by providing a username and get back the user's (hashed) password, so that Shiro can validate a user's username and password.

ShiroAuthorizingRealm.java (link)

public class ShiroAuthorizingRealm extends AuthorizingRealm {

    public ShiroAuthorizingRealm(){
        setName("ShiroAuthorizingRealm");
        setCredentialsMatcher(
            new HashedCredentialsMatcher(Sha1Hash.ALGORITHM_NAME));
    }

    @Override
    public AuthenticationInfo doGetAuthenticationInfo(
        AuthenticationToken authToken) throws AuthenticationException {

        UsernamePasswordToken token = (UsernamePasswordToken) authToken;

        User user;
        try {
            user = loadUserByUsername( token.getUsername() );

        } catch (WebloggerException ex) {
            throw new AuthenticationException(
                "Error looking up user " + token.getUsername(), ex);
        }

        if (user != null) {
            return new SimpleAuthenticationInfo( 
                user.getUserName(), user.getPassword(), getName());

        } else {
            throw new AuthenticationException(
                "Username not found: " + token.getUsername());
        }
    }

In the code above you can see how we pull the username out of the authToken provided by Shiro and we call a method, loadUserByUserName(), which uses Roller's Java API to load a Roller user object specified by name.

The next method of interest is doGetAuthorizationInfo(), which allows Shiro to look-up a user's Role. This allows Shiro to detmerine if the user is a Roller admin user or a blog editor.

ShiroAuthorizingRealm.java (continued)


    public AuthorizationInfo doGetAuthorizationInfo(
        PrincipalCollection principals) {

        String userName = (String)
            (principals.fromRealm(getName()).iterator().next());

        User user;
        try {
            user = loadUserByUsername( userName );
        } catch (WebloggerException ex) {
            throw new RuntimeException("Error looking up user " + userName, ex);
        }

        Weblogger roller = WebloggerFactory.getWeblogger();
        UserManager umgr = roller.getUserManager();

        if (user != null) {
            List roles;
            try {
                roles = umgr.getRoles(user);
            } catch (WebloggerException ex) {
                throw new RuntimeException(
                    "Error looking up roles for user " + userName, ex);
            }
            SimpleAuthorizationInfo info = new SimpleAuthorizationInfo();
            for ( String role : roles ) {
                info.addRole( role );
            }
            log.debug("Returning " + roles.size() 
                + " roles for user " + userName + " roles= " + roles);
            return info;

        } else {
            throw new RuntimeException("Username not found: " + userName);
        }
    }

In the code above you can see that we use the loadUserByUsername() too look-up a user by username, then we use Roller's Java API to get the user's roles. We add those roles to an instance of the Shiro class SimpleAuthorizationInfo and return it to Shir.

Creating a Shiro Authorizing Filter

Now that we've implementated a realm, we've provided Shiro with everything needed to authenticate Roller users and get access to Roller user role information. Next, we need to configure Shiro to enforce roles for the URL apths found in Roller. Shiro includes a RolesAuthorizationFilter, which is close to what we need but not exactly right for Roller. I had to extend Shiro's roles filter so that we can allow a user who has any (not all) of the required roles for a resource.

RollerRolesAuthorizationFilter.java (link)

public class RollerRolesAuthorizationFilter 
    extends RolesAuthorizationFilter {

    @Override
    public boolean isAccessAllowed( 
        ServletRequest request, 
        ServletResponse response, 
        Object mappedValue) throws IOException {

        final Subject subject = getSubject(request, response);
        final String[] roles = (String[]) mappedValue;

        if (roles == null || roles.length == 0) {
            return true;
        }

        // user is authorized if they have ANY of the roles
        for (String role : roles) {
            if (subject.hasRole(role)) {
                return true;
            }
        }
        return false;
    }
}

Configuring Shiro for Roller

Now that we've seen the Java code needed to hook Shiro into Roller, lets look at how we configure Shiro to use that code. We do that using the Shiro configuration file: shiro.ini, as shown below.

shiro.ini (link)

[main]

defaultRealm = org.apache.roller.weblogger.auth.ShiroAuthorizingRealm
securityManager.realms = $defaultRealm

cacheManager = org.apache.shiro.cache.ehcache.EhCacheManager
securityManager.cacheManager = $cacheManager

authc = org.apache.shiro.web.filter.authc.FormAuthenticationFilter
authc.loginUrl = /roller-ui/login.rol
authc.successUrl = /roller-ui/menu.rol

rollerroles = org.apache.roller.weblogger.rest.auth.RollerRolesAuthorizationFilter

[urls]

/roller-ui/login.rol          = authc
/roller-ui/login-redirect.rol = authc, rollerroles[admin,editor]
/roller-ui/profile**          = authc, rollerroles[admin,editor]
/roller-ui/createWeblog**     = authc, rollerroles[admin,editor]
/roller-ui/menu**             = authc, rollerroles[admin,editor]
/roller-ui/authoring/**       = authc, rollerroles[admin,editor]
/roller-ui/admin/**           = authc, rollerroles[admin]
/rewrite-status/**            = authc, rollerroles[admin]
/roller-services/rest/**      = authcBasic, rollerroles[admin,editor]

In the configuration file above, you see how we hook in the new ShiroAuthorizingRealm on line 3. The next couple lines are boiler-plate code to hook in Shiro's caching mechanism and then, on line 9, we configure an authentication method called authc, which is configured to use Shiro's Form Authentication feature. And, on line 13, we hook in our new RollerRolesAuthorizationFilter.

Next, we tell Shiro that the login page for Roller is /roller-ui/login.rol and which page to direct a user to on a successful login, /roller-ui/menu.rol, if the user did not specify which page they wanted to access. And finally, on lines 17-25, you see the list of Roller URL patterns that need protection, which authentication method to use (authc or authcBasic) and the authorization filter and roles required for access to the URL pattern.

Wrapping up...

That's all there is to the story of Roller and Shiro so far. I was able to get Roller's form-based authentication working with Shiro, but I did not try to test with OpenID or LDAP, so I assume more work will be necessary to get them working. I did the work in my experimental Rollarcus fork of Roller. You can get the code from the shiro_not_spring branch. Pull requests are quite welcome as are suggestions for improvement. Please let me know if you see anything wrong in the above code.

This work may not find its way into Roller proper, but it plays a part in my the next side-project that I will share: A REST API for Roller with JAX-RS.

Usergrid and Ember.js - part 2

2015-01-26T10:23:01+00:00

In part one, I explained the basics of the example Usergrid-Ember "Checkin" app, how the index page is displayed and how login is implemented. In part two, I'll explain how Ember.js can be hooked into the Usergrid REST API to store and query JSON objects.

Ember.js includes a feature referred to as Ember-Data, which provides a persistence interface for storing and retrieving JavaScript objects that could be stored in memory, or stored on a server and accessed via REST API.

To use Ember-Data with your REST API you've got to define an Ember-Data model and add an Ember-Data REST adapter. If your REST API differs from what Ember-Data expects then you will probably have to extend the built-in REST adapter to handle your URL pattens, and extend the built-in REST serializer to handle your JSON format. By extending Ember-Data in this way, you can use it to store and query data from Usergrid without using the Usergrid JavaScript SDK at all. Below I'll explain what I had to do to make the Checkin app's Activities collection available via Ember-Data.

Define Ember-Data models

Ember-Data expects each of your REST API collections to have a defined data model, one that extends the DS.Model class. Here's what I added for the Activities collection:

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L18" >link)

App.Activity = DS.Model.extend({
  uuid: DS.attr('string'),
  type: DS.attr('string'),
  content: DS.attr('string'),
  location: DS.attr('string'),
  created: DS.attr('date'),
  modified: DS.attr('date'),
  actor: DS.attr('string'),
  verb: DS.attr('string'),
  published: DS.attr('date'),
  metadata: DS.attr('string')
});

Create a custom RESTAdapter

The Ember-Data REST adapter expects a REST API to follow some common conventions for URL patterns and for JSON data formats. For example, if your REST API provides a collection of cats then Ember-Data will expect your REST API to work like so:

What Ember-Data expects for a cats collection:

GET /cats - get collection of cats
POST /cats - create new cat.
GET /cats/{cat-id} - get cat specified by ID.
PUT /cats/{cat-id} - update cat specified by ID.
DELETE /cats/{cat-id} - delete cat specified by ID.

Usergrid follows the above conventions for collections, but there are some exceptions. For example, the Usergrid Activities collection. A GET on the /activities path will return the Activities of the users that you (i.e. the currently authenticated user) follow. You don't POST new activities there, instead you post to your own Activities collection at the path /users/{your-user-id}/activities. It works like this:

Usergrid's Activities collection:

GET /activities - get Activities of all users that you follow.
POST /user/{user-id}/activities - create new Activity for user specified by ID
GET /user/{user-id}/activities - get Activities for one specific user.

To adapt the Activities collection to Ember-Data, I decided to create a new model called NewActivity. A NewActivity represents the data needed to create a new Activity, here's the model:

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L132" >Link)

// Must have a special model for new activity because new 
// Activities must be posted to the path /{org}/{app}/users/activities, 
// instead of the path /{org}/{app}/activities as Ember-Data expects.
App.NewActivity = DS.Model.extend({
  content: DS.attr('string'),
  location: DS.attr('string'),
  actor: DS.attr('string'),
  verb: DS.attr('string')
});

Then, in Checkin's custom REST adapter, I added logic to the pathForType() function to ensure that NewActivities are posted to the correct path. Here's the adapter:

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L40" >Link)

App.ApplicationAdapter = DS.RESTAdapter.extend({

  host: Usergrid.getAppUrl(),

  headers: function() { 
    if ( localStorage.getItem("access_token") ) {
      return { "Authorization": "Bearer " 
          + localStorage.getItem("access_token") }; 
    } 
    return {};
  }.property().volatile(), // ensure value not cached

  pathForType: function(type) {
    var ret = Ember.String.camelize(type);
    ret = Ember.String.pluralize(ret);

    if ( ret == "newActivities" ) {
      // Must have a special logic here for new activity 
      // because new Activities must be posted to the 
      // path /{org}/{app}/users/activities, instead of the 
      // path /{org}/{app}/activities as Ember-Data expects.
      ret = "/users/" + Usergrid.user.username + "/activities";
    }
    return ret;
  }

});

You can see a couple of other interesting things in the example above. First, there's the host field which specifies the base-URL of the REST API for the Checkin app. Next, there's the headers() function, which ensures that every request carries the access_token that was acquired during login.

Create a custom RESTSerializer

Ember-Data also has expectations about the JSON format returned by a REST API. Unfortunately, what Ember-Data expects and what Usergrid provides are quite different. The two examples below illustrate the differences:

Ember-Data vs. Usergrid JSON formats

Ember-Data expects collections like this:

{
   cats: [{
       "id": "6b2360d0",
       "name": "enzo",
       "color": "orange"
   },{
       "id": "a01dfaa0",
       "name": "bertha",
       "color": "tabby"
   }]
}

Usergrid returns collections like this:

{
   action: "get",
   path: "/cats",
   count: 2,
   entities: [{
       "uuid": "6b2360d0",
       "type": "cat",
       "name": "enzo",
       "color": "orange"
   },{
       "uuid": "a01dfaa1",
       "type": "cat",
       "name": "bertha",
       "color": "tabby"
   }]
}

Ember-Data expects individual objects like this:

 
{
   cat: {
       "id": "a01dfaa0",
       "name": "bertha",
       "color": "tabby"
   }
}

Usergrid returns individual objects like this:

 
{
   "id": "a01dfaa0",
   "type": "cat",
   "name": "bertha",
   "color": "tabby"
}

You can see two differences above. Ember-Data expects JSON objects to be returned with a "type key" which you can see above: the "cats" field in the collection and the "cat" field in the individual object. Also, Ember-Data expects an object's ID field to be named "id" but Usergrid returns it as "uuid."

The deal with these differences, the Checkin app extends Ember-Data's DS.RESTSerializer. Here's the code:

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L75" >Link)

App.ApplicationSerializer = DS.RESTSerializer.extend({

  // Extract Ember-Data array from Usergrid response
  extractArray: function(store, type, payload) {

    // Difference: Usergrid does not return wrapper object with 
    // type-key. So here we grab the Usergrid Entities and stick 
    // them under a type-key
    var typeKey = payload.path.substring(1);
    payload[ typeKey ] = payload.entities;

    // Difference: Usergrid returns ID in 'uuid' field, Ember-Data 
    // expects 'id'. So here we add an 'id' field for each Entity, 
    // with its 'uuid' value.
    for ( var i in payload.entities ) {
      if ( payload.entities[i] && payload.entities[i].uuid ) {
        payload.entities[i].id = payload.entities[i].uuid;
      }
    }
    return this._super(store, type, payload);
  },

  // Serialize Ember-Data object to Usergrid compatible JSON format
  serializeIntoHash: function( hash, type, record, options ) {

    // Usergrid does not expect a type-key
    record.eachAttribute(function( name, meta ) {
      hash[name] = record.get(name);
    });

    return hash;
  }
});

In the code above you can see how the extractArray() method moves the "entities" collection returned by Usergrid into a type-key field as expected by Ember-Data and how it copies the "uuid" field to add the "id" field that Ember-Data expects.

We also need to transform the data that Ember-Data sends to Usergrid. You can see this above in the serializeInHash() function, which ensures that when data is POSTed or PUT to Usergrid, the type key is removed because that's what Usergrid expects.

Implementing Add-Checkin

To implement Add-Checkin, I added an HTML template called "add-checkin" to Checkin's index.html file. The template displays an Add-Checkin form with two fields: one for content and one for the location. Here's what it looks like in all its modal glory:

Both fields are simple strings (someday I'd like to extend Checkin to use location information from the browser). I won't go into detail here, but it took a bit of research to figure out how to make a Bootstrap modal dialog work with Ember.js. Below you can see the add-checkin controller, which provides a save() function to save a new checkin.

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L390" >Link)

App.AddCheckinModalController = Ember.ObjectController.extend({

  actions: {

    save: function( inputs ) {

      var content = inputs.content;
      var location = inputs.location;
      var target = this.get("target");

      var activity = this.store.createRecord( "NewActivity", {
        content: content,
        location: location,
        verb: "checkin",
        actor: {
          username: Usergrid.user.username
        }
      });

      activity.save().then(
        function( success ) { 
          alert("Saved"); 
        },
        function( error ) { 
          alert("Error " + error.responseJSON.error_description); 
        }
      ); 

    } 
  }
});

In the code above you can see how easy it is to access Usergrid data via Ember-Data now that we've got our custom REST adapter and serializer in place. We create a new Activity with a call to this.store.createRecord() and to save it all we need to do is activity.save().

Time to wrap up...

To sum things up, here are some closing thoughts and observations.

If you are considering JavaScript MVC frameworks, then Ember.js is definitely worthy of your consideration. The documentation makes it easy to learn and the community is friendly and helpful.
It would be great for Usergrid to provide an Ember.js SDK that makes it really easy to build apps with Ember.js and Usergrid.
Ember-Data is an integral part of Ember.js, something that you need to do pretty much anything, but it is treated as a separate package with separate documentation. That is somewhat confusing for a new user.
Ember-Data does not include built-in form validation so if your app includes a large number of non-trivial forms, then you may prefer AngularJS over Ember.js.
There is a form validation plugin for Ember.js, but it requires the experimental Ember-CLI utility. I tried to use it, but Ember-CLI was unpleasnt enough that I gave up.

I appreciate any feedback you might have about this article, the Usergrid-Ember project and Apache Usergrid. If you want to see how the whole Usergrid-Ember project fits together, find it on GitHub here: Usergrid-Ember. Next up, I'll write about my experiences using Apache Shiro to replace Spring Security in Apache Roller.

Usergrid and Ember.js - part 1

2015-01-20T09:39:47+00:00

The next one of my 2014 Side projects that I’d like to share is Usergrid-Ember, an experiment and attempt to learn more about Ember.js and Apache Usergrid by implementing the Checkin example from my Usergrid mobile development talk. If you're interested in either Usergrid or JavaScript web development then I hope you'll read on...

Why Ember.js?

Ember.js is one of the leading frameworks for building browser-based apps. It's one of many JavaScript Model View Controller (MVC) frameworks. Generally speaking, these frameworks let you define a set of routes or paths in your app, for example /index, /orders, /about, etc. and map each to some JavaScript code and HTML templates. Handling a route usually means using Ajax to grab some “model” data from a server and using a template to create an HTML “view” of the data that calls functions provided in a "controller" object.

JavaScript MVC frameworks are not simple and each has its own learning curve. Is it really worth the learning time when you can do so much with a little library like jQuery? For most projects I think the answer is yes. These frameworks force you to organize your code in a logical and consistent way, which is really important as projects grow larger, and they provide features that may save you a lot of development time.

Based on what I've seen on the net and local meet-ups, the leading frameworks these days are Ember.js and AngularJS. After I saw Yehudi Katz’s talk at All Things Open, I decided to spend some time learning Ember.js.

Getting started with Ember.js

The first thing you see when you visit the Ember.js site is a big button that says "DOWNLOAD THE STARTER KIT" and so that is where I started. The Starter Kit is a, a minimal Ember.js project with about twenty JavaScript, HTML and CSS files. It's a good way to start: small and simple.

Ember.js Starter Kit files:

Sidebar: I do hope they keep the Starter Kit around when the new Ember-CLI tool matures. Ember-CLI generates too many magic boiler-plate files and sub-directories for somebody who is trying to understand the basics of the framework. And this is an interesting point of view: Ember-CLI is Making You Stupid by Yoni Yechezkel.

Other stuff: Bower, Grunt and Bootstrap

I like to bite off more than I can chew, so I decided to use a couple of other tools. I used Bower to manage dependencies and Grunt to concatenate and minify those dependencies, and other things like launching a simple web server for development purposes. I also decided to use Bootstrap to provide various UI components needed, like a navbar and nicely styled list views.

I won't cover the details, but it was relatively easy to get Bower and Grunt working. Here are the config files in case you are interested: bower.json and Gruntfile.js. I did hit one problem: when I included Bootstrap as one of my dependencies the Glyphicons would all appear as tiny boxes, so I decided to pull Bootstrap from a CDN instead (looks like there is a fix for that now).

Defining Index Route, Model and Template

Every Ember.js app needs to define some routes. There is a default route for the "/" path which is called the index route, and you can add your own routes using the Router object. The snippet below shows what I needed to get started:

Part of app.js (link)

// create the ember app object
App = Ember.Application.create();

// define routes
App.Router.map(function() {
    this.route("login", { path: "/login" });  
    this.route("logout", { path: "/logout" });
    this.route("register", { path: "/register" });
});

Ember.js will look for the JavaScript Route and Controller objects as well as the HTML template using the names above. For example: Ember.js will expect the login route to be named App.LoginRoute, the controller to be named App.LoginController and the template to be named "login."

Let's talk about the index route. When a user arrives at your app they’ll be directed to the index route. Ember.js will then look for a JavaScript object called App.IndexRoute to provide the model data and JavaScript functions needed for the index page. Here’s a partial view of the index route:

Part of app.js (link)

App.IndexRoute = Ember.Route.extend( {

    // provide model data needed for index template
    model: function() {
        if ( this.loggedIn() ) {
            return this.store.find("activity");
        }
        return [];
    }

});

The index page of the Checkin app shows the Checkin activities of the people that you follow. Above you can see how to route's model() function makes that data available to the template for display. If the user is logged in we call the store.find(“activity”) function to call the Usergrid REST API to get an array of the latest Activity objects. There is some serious Ember-Data magic going on there and I'll cover that in part two of this article.

To display the index route, Ember looks for an HTML template called “index” and will use that template to display the index page. Below is the index template. The template is a Handlebars template and the things that appear in double curly-braces are Handlebars expressions.

Part of index.html (link)



  
    

      
        Checkin
      

      
          {{action 'showModal' 'add-checkin-modal' model }}>Add Checkin
      

      
        Logout 
      

    
  

  
    

      
        {{#each item in model}}
          
              {{item.content}} | {{item.location}}
          
        {{/each}}

In the above template you can see a couple of {{action}} expressions that call out to JavaScript methods defined in the Checkin app. The part of the code that uses the model is in the {{#each}} loop which loops through each Activity in the model and dispays an HTML list with the the item.content and item.location of each Activity.

Here's what the above template looks like when displayed in a browser:

Implementing Login

In Checkin, login is implemented using HTML Local Storage. Once a user has successfully logged in, the app stores the username and the user's access_token in Local Storage. When user arrives at the index page, we check Local Storage to see if that user is logged in and if not, we direct them to the login route, which in turn displays the login page using the template below.

Part of index.html (link)



  
    
      
        Checkin
      
    
  

  
    
      

        

         Please Login

         Email address
         {{input class="form-control" type="text" 
              valueBinding="username" placeholder="Username"}}

         Password
         {{input class="form-control" type="password" 
             valueBinding="password" placeholder="Password"}}
           
         
            Login
         

         Register as new user.

The LoginController provides the functions needed by the Login page itself and there are two. There is a login() function (called on line 27 above) that performs the login, and there is a register() function (called on line 31 above) that directs the user to the New User Registration page. Here's a snippet of code from the App.LoginController that provides these two functions:

Part of app.js (link)

App.LoginController = Ember.Controller.extend({

  actions: {

    login: function() { 

      // login by POST to Usergrid app's /token end-point

      var loginData = {
        grant_type: "password",
        username: this.get("username"),
        password: this.get("password")
      };

      $.ajax({
        type: "POST",
        url: Usergrid.getAppUrl() + "/token",
        data: loginData,
        context: this,
        error: function( data ) {

          // login failed, show error message
          alert( data.responseJSON.error_description );
        },
        success: function( data ) { 

          // store access_token in local storage
          Usergrid.user = data.user;
          localStorage.setItem("username", loginData.username );
          localStorage.setItem("access_token", data.access_token );

          // clear the form
          this.set("username", ""); 
          this.set("password", "");

         // call route to handle post-login transition
          this.get("target").send("onLogin"); 
        }
      });
    },

    register: function() {
      this.transitionToRoute("register");
    }

  }
});

The above code shows how to login to a Usergrid app using jQuery's Ajax feature. The login() function takes the username and password values from the login form, puts those in a JSON object with grant_type "password" and posts that object to the /token end-point of the Usergrid app. If that post succeeds, the response will include an access_token. We store that in Local Storage; we'll need to use it in all subsequent calls to Usergrid.

Usergrid fans will notice that I'm not using the Usergrid JavaScript SDK. That's because Ember.js provides Ember-Data, which acts as a very nice REST client and can be adapted to work with the URL structure and JSON formats of just about any REST API. I'll write about that in part two of this article.

2014 side projects

2015-01-05T08:37:04+00:00

For various reasons, I've always got a couple of coding projects on the back burner, things that I hack around with on weekends and breaks. In 2014, I started four projects and learned about Ember.js, jQuery Mobile, Apache Shiro, Apache CXF and the Arquillian test framework.

I like to share my code, so I've put my code on GitHub and I'm going to write a brief post about each here on my blog. I'll provide links as I go and, of course, I welcome any criticisms and suggestions for improvement that you might have. First up: the Usergrid-Mobile project.

The Usergrid-Mobile project

To be honest, Budapest was the goal of this project. In the Spring of 2014, I decided that the best chance of getting to ApacheCon EU in Budapest was to create a great "mobile development with Usergrid" talk, and to do that I needed a great example project. The resulting project shows how to create a dumbed-down Foursquare-style "checkin" app using HTML5, JavaScript, jQuery Mobile and Apache Cordova.

Luckily for me, my talk was accepted for ApacheCon EU and in November I traveled to Budapest (took some photos) and gave the talk there.

I also presented the talk at the All Things Open conference in Raleigh, NC and you can view a video of that talk, Mobile Development with Usergrid on YouTube.

You can find the code for usergrid-mobile on GitHub. I also created a Vagrant File to launch a local instance of Usergrid for demo purposes. It's called usergrid-vagrant.

That's all for now. Next up: Usergrid-Ember.

Phoenix Websites

2014-12-01T07:07:25+00:00

My eldest son Alex and his friend Austin have started a website design and creation business called Phoenix Websites and, of course, I think this is a great thing. They're not yet out of high school and just getting started, but they've already landed a couple of real live customers. They've got some skills and are not afraid of hard work, so if you're a Triangle-area small business owner and you need a nice new website, check them out.

Like any new business, they need some link love so here we go: Phoenix Websites: Website design and creation services in the Raleigh-Durham Triangle-area. Follow them on twitter at @phoenixrdu

Introduction to Apache Usergrid

2014-11-29T10:09:48+00:00

I travelled to Budapest, Hungary for a couple of weeks for a very nice vacation with my wife and to speak at ApacheCon EU. Here are the slides that I presented at ApacheCon EU:

(you can also view the presentation on Slideshare.)

And here is the session abstract:

Whether you are building a mobile app or a web app, Apache Usergrid (incubating) can provide you with a complete backend that supports authentication, persistence and social features like activities and followers all via a comprehensive REST API — and backed by Cassandra, giving you linear scalability. All that, and Usergrid is open source too.

This session will explain how you can use Usergrid to provide a back-end for your application. We’ll start with an overview of Usergrid features, then explore in depth how to authenticate users, store data and query data with the REST API provided by a Usergrid server. We’ll develop a simple HTML5 app and package it as a native mobile app via Apache Cordova. We'll also cover how to run Usergrid locally for development and testing.

Talking Usergrid at ApacheCon 2014

2014-03-16T16:10:16+00:00

I've been working at Apigee since September 2013 and one of the things I love most about my new job is the fact that I'm actively contributing to open source again.

I'm working on Apache Usergrid (incubating), an open source Backend-As-A-Service (BaaS) that's built on the Apache Cassandra database system. Apigee uses Usergrid as part of Apigee Edge (see the Build Apps part of the docs).

Apigee contributed code for Usergrid to the Apache Software Foundation back in October 2013 and Usergrid is now part of the Apache Incubator. The project is working towards graduating from the Incubator. That means learning the Apache way, following the processes to get a release out and most importantly, building a diverse community of contributors to build and maintain Usergrid.

One on the most important parts of building an open source community is making it easy for people to contribute and and that's why I submitted a talk to the ApacheCon US 2014 conference (April 7-9 in Denver, CO) titled How to Contribute to Usergrid.

The talk is intended to be a briefing for contributors, one that will lead you through building and running Usergrid locally, understanding the code-base and test infrastructure and how to get your code accepted into the Usergrid project.

Here's the outline I have so far:

How to Contribute to Apache Usergrid

Motivation
- Why would anybody want to contribute to Usergrid?
First steps
- The basics
- Getting signed up
Contributing to the Stack
- Understanding the architecture & code base
- Building the code. Making and testing changes
- Running Usergrid locally via launcher & via Tomcat
Contributing to the Portal
- Understanding the architecture & code base
- Building the code. Making and testing changes
- Running the portal locally via node.js
Contributing to the SDKs
- Understanding the architecture & code base
- Building the code. Making and testing changes
Contributor workflow: how to get your code into Usergrid
- For quickie drive-by code contributions
- For more substantial code contributions
- For documentation & website changes
Contributing Docs and Website changes
- Website, wiki and GitHub pages
- How to build the website and docs
Roadmap
- First release
- New Core Persistence system
- The two-dot-o branch
- Other ideas

I'm in the process of writing this talk now so suggestions and other feedback are most welcome.

How Wayin does Cloud Orchestration

2013-06-18T16:16:34+00:00

I've already mentioned this on Twitter and LinkedIn, but just in case you missed it: I'll be speaking tomorrow night at the Triangle AWS and Triangle DevOps joint meetup at Argyle Social in Durham, NC. I'll give a quick overview of cloud orchestration and Wayin Hub. Then I'll dive into the details of how we automate deployment, scaling and backups for Wayin Hub using AWS and AWS Cloud Formation.

As a little teaser, here's a GIF animation of my automated deployment slide:

For more information check the Triangle DevOps page for the event.

UPDATE: slide are now on SlideShare.

Composite Keys in Apache Cassandra

2013-05-23T17:56:49+00:00

Newer versions of Apache Cassandra include CQL, an SQL-like query language that supports both query, update and delete statements as well as the Data Definition Language (DDL) statements like create and alter for tables and indexes.

You can create tables (known as column families in Cassandra lingo) just like you can in a relational database, but there are some caveats. One caveat is this: if you want to sort and use ORDER BY in queries on a table, then you will have to use composite-key in that table and that composite key must include the field that you wish to sort on. You have to decide which fields you wish to sort on when you design your data model, not when you formulate queries. Another caveat is that, with Cassandra 1.1, there is no support for secondary indexes on composite-keyed tables. That means you can only query on the fields in the composite-key and in certain specific ways. More on that later.

In version 1.1, Cassandra supports (at least) two different models for storing data. You can use a single primary key in your table, or you can use a composite key. Cassandra stores your data differently for these two cases and the queries that you can perform on these two types of tables vary as well.

I’ll explain how simple-keyed and composite-keyed differ with some examples. The example is the Bite table, which holds a chunk of data identified by an ID and sorted by an integer score value. I don’t want to get into details, but in the product I'm working on, Wayin Hub, a Site contains Feeds and Feeds contain Bites, which might represent Tweets, RSS items or other social network activities. Hopefully, that will be enough background for you to understand the examples below.

Single-keyed Table

With a single keyed table you have a row for each entity that you store and a column in that row for each field of the entity. For example, to represent the Bite table as a single-keyed table it would be defined like so:


   create table bite (
	id varchar PRIMARY KEY,
	feedid varchar,
	score bigint,
	data varchar
   );

   create index bite_feedid on bite (feedid);
   create index bite_score on bite (score);

We need those id, feedid and score fields so we can look up bites by those values. The data field is used to store a JSON representation of other data we associate with each Bite. For example, if you only have three Bites in the table, here’s what the results of a select * from bite might look like:


 id   |  feedid  |  score  | data
------+----------+---------+----------------------
bite2 |  feed0   |   101   | { “status” : “APPROVED”, ... 
bite3 |  feed0   |   102   | { “status” : “DENIED”, ...
bite1 |  feed0   |   100   | { “status” : “APPROVED”, ...

The way the data is stored in Cassandra would look about the same, as illustrated in the diagram below. Each table row corresponds to a Row in Cassandra, the id of the table row is the Cassandra Row Key for the row. Each value in the row is a Cassandra Column with a key and a value. If you add more table rows, you get more Cassandra Rows.

Now let’s get back to the topic of this post and that caveat that I mentioned earlier.

Composite-keyed Table

If we want to sort data in a table, then we need to use a composite-keyed table. With a composite-keyed table you define a composite-key made up of multiple fields from the table. The first key is known as the partition-key. To sort by score we also include the score in the composite key. And since it is possible for two Bites with the same partition key to occur at the very same time, we also include varchar ID to ensure uniqueness.

Here’s how the Bite table is defined. The composite-key is the list of three fields in PRIMARY KEY parentheses.


   create table Bite (
	 partkey varchar,
	 score bigint,
	 id varchar,
	 data varchar,
	 PRIMARY KEY (partkey, score, id)
   ) with clustering order by (score desc);

And if you had three Bites in the table the query select * from bite would return this:


 partkey  |  score  |  id   |  data
----------+---------+-------+----------------------
  feed0   |   102   | bite3 | { “id” : “bite2”, ... 
  feed0   |   101   | bite2 | { “id” : “bite3”, ...
  feed0   |   100   | bite1 | { “id” : “bite1”, ...

The surprise is how this table is stored in Cassandra. Instead of storing a Cassandra Row for each table row, the data is stored as one row. The more Bites you add to the table, the more Cassandra Columns are added to that Row. The diagram below illustrates how this works for the three Bites of data above. There is one Row with key of feed0. And there is one Column for each table row of data. Each Column uses a key that combines the score and id plus a string that indicates what field is stored in the Column Value, which in our case is only the data field.

Querying a Composite-keyed Table

With Cassandra 1.1, if you want to select a single Bite, you must know all of the composite-keys. Here’s an example query that selects a single Bite:


   select data from bite where partkey=’feed0’ and score=101 and id=’bite2’

To get latest Bites in a Site’s Feed, you specify only the partition-key and ask for ordering by score, like so:


   select data from bite where partkey=’feed0’ order by score desc limit 20

If you try to query without specifying the partition key and the score, you will get an error message. For example, this query:


   select data from bite where id=’bite2’ limit 20

Would give you this error message:


   Bad Request: PRIMARY KEY part id cannot be restricted (preceding
   part score is either not restricted or by a non-EQ relation)

That means we can’t look up Bites by a single ID. That’s not very convenient but that’s the way it is with Cassandra 1.1 which does not allow additional indexes on composite-key tables.. If you really want to lookup Bites by id, you have to create an entirely new simple-keyed table with Bite id as the primary key and use that table to look up the a Bite’s partKey and score. This problem is fixed in Cassandra 1.2 because it allows secondary indexes on fields in composite-key table. Let’s talk about that.

Secondary Indexes

Cassandra 1.2 comes with support for secondary indexes on composite-keyed tables, but you cannot create a secondary index on keys that are already part of the composite-key. So, if we want to be able to look-up Bites by ID, then we must add a second and redundant biteid field like so:


   alter table Bite add biteId varchar;
   create index Bite_biteId on Bite (biteId);

Inside Cassandra, the the data would look like this, a new field in the table means a new Column in the row, as show below:

And with that secondary index we can support queries like this:


   select data from bite where biteid=’bite2’

Cassandra 1.2

In the DataStax Cassandra 1.2 docs, it says “CQL3 transposes data partitions (sometimes called "wide rows") into familiar row-based result sets, dramatically simplifying data modeling.

I believe that means that all tables are stored the way that composite-keyed tables are stored. According to the docs, legacy tables from Cassandra 1.1 are supported in 1.2 and, if you want, you can still create Cassandra 1.1 style tables by using the “compact storage” attribute. For example, to create the Bite table with a the Cassandra 1.1 table model and a single primary key you’d do this:


   create table bite (
	id varchar PRIMARY KEY,
	feedid varchar,
	score bigint,
	data varchar
   ) with compact storage;

And that’s all I’ve got on this topic.

Wrapping up...

I wrote this up to help myself understand how composite-keyed tables work in Cassandra, so I’d love any feedback you might have and especially if you think I’ve got concepts or terminology wrong. Thanks for reading.

You can read more about Cassandra 1.1 tables on the Datastax site:
http://www.datastax.com/docs/1.1/ddl/column_family

More about Cassandra 1.2 tables:
http://www.datastax.com/docs/1.2/ddl/table

More about Cassandra 1.1 legacy tables in Cassandra 1.2
http://www.datastax.com/docs/1.2/ddl/legacy_table

Also, I found these posts by Brian O’Neill very helpful:
http://brianoneill.blogspot.com/2012/09/composite-keys-connecting-dots-between.html
http://brianoneill.blogspot.com/2012/10/cql-astyanax-and-compoundcomposite-keys.html

And, if you want to build an engaging site for your customers, fans or constituents based on live tweets, photos and videos check out Wayin Hub and follow @wayinhub on Twitter.

WIP: Widgets and Gadgets

2012-02-22T07:52:11+00:00

This is the fifth in my series of Web Integration Patterns. Check out the intro at this URL http://rollerweblogger.org/roller/entry/web_integration_patterns

Synopsis

Allow other web sites and applications to integrate your site into their web pages by providing an embeddable user interface, commonly known as a Gadget or Widget, which allows users to view and interact with your site in the context of other sites.

Motivations

By embedding Widgets in your site, you can make your site more useful and informative to your users. Users can access relevant information from other sites in the context of your web site.
By allow other sites to embed your Widgets, you can give your site and the services that it offers wider reach. Your users can access and interact with your services in the context of other sites.

Related Patterns

Dashboards and Portals
Social Network Plugins
Delegated Pickers

A Widget or Gadget is a piece of an application, a visual component or a chunk of user-interface that can be added to some other application. On the web, Widgets are implemented using HTML and JavaScript, or sometimes Flash. Here we'll discuss simple Widgets and leave discussion or more advanced usage of Widgets in Dashboards, Portals and Pickers for the more advanced patterns we'll discuss later.

Easy to "consume"

Widgets are easy, especially for widget "consumers" i.e. those who wish to add an existing Widget to their site. Adding a widget to your site or web application can be as easy as adding a couple of lines of HTML and JavaScript to the pages of your site. You can see some examples of Widgets in the two screen shots I've included here, below is the Wayin polling Widget, which shows my latest Wayins:

And below is the Flickr Widget, which shows a random selection of my photos from Flickr. You can see these two Widgets in action on the front page of my blog. And, by the way, the Like, Tweet and +1 buttons on the right of this entry (when viewed on my blog site) are also Widgets.

Relatively easy to create

Developing an entirely new Widget more complicated than using one that already exists. Creating a new Widget typically requires writing HTML and JavaScript code and employing some interesting tricks. The simplest way to implement a Widget is to use an iframe, allowing a portion of one site to be embedded in another site.

I'm not going to go into detail as there are already so many good resources on the web for learning about Widget crafting. For example, Alex Marandon wrote a pretty comprehensive blog post on creating Widgets covering basic issues like how to ensure that your widget code does not interfere with the page hosting the Widget, how to dynamically load resources and how to bypass browsers’ single-origin policy using JSON-P.

Widget and Gadget standards

For simple Widgets you don't really need any standards other than HTML and JavaScript. If you want to target specific web site Dashboards or Portals, like NetVibes, or enterprise web applications like Jive or Atlassian's software development tools, then you'll need to create your Widgets with specific APIs in mind. For example, NetVibes provides it's own Widget API, while Jive and Atlassian support the Open Social Gadget APIs. I'll talk more about these issues when I cover the related patterns Dashboards, Portals and Social Network Plugins.

Wrapping up...

That's it for Widgets and Gadgets. Next up: Feed-based Integration.

Rollarcus: from 9 to 2

2011-12-20T07:56:37+00:00

I made some progress in Rollarcus over the past couple of weekends, but not a lot. This makes me wonder how I ever found the "nights and weekends" to get Roller started in the first place, but that's a different topic.

What I've done so far in Rollarcus is to simplify things. While I was at Sun, we split Roller up into a number of parts: a weblogger part for blogging, a planet part for RSS/Atom aggregation, a core part for things common to both. After Sun, I worked to move Roller to Maven and further split things up into a total of 9 Maven modules including an assembly for building the release. Now, I think that all these modules are unnecessary -- we never shipped a Roller-Planet application and nobody wants to use parts of Roller -- and even if they did, the modules did not really help.

Here's the before view: apache/roller

Here's the after view: snoopdave/rollarcus

So, in Rollerarcus, I've merged all the modules. Except for one "test utilities" module, all Java code, JSPs and other code is now in one module and much easier to deal with. Next, I'm going to attack the (what I consider to be) unnecessary dependencies and drastically reduce the number of jars in WEB-INF.

UPDATE: The most significant of the changes that I made in Rollarcus have been applied by to Apache Roller and today (August 18, 2013) I removed the Rollarcus repository from Github.

Fork it all

2011-12-03T11:08:29+00:00

I just forked Roller on Github.

The new project is called Rollarcus and is mostly just an experiment and, I hope, a learning experience. I've got some ideas about stripping Roller down to it's core and making it more fun and easy to develop and deploy. We'll see how far I get.

In case you don't already know, an arcus cloud is a low, horizontal cloud formation.

UPDATE: The most significant of the changes that I made in Rollarcus have been applied by to Apache Roller and today (August 18, 2013) I removed the Rollarcus repository from Github.

Upgraded to Roller 5.1-dev

2011-11-26T14:19:28+00:00

I just upgraded this site to Roller 5.1-dev, Subversion rev 1175172. This unreleased version of Roller includes new mobile theming capability (mentioned in my previous post), allowing a theme to define both a standard and a "mobile" version of each weblog page. I haven't added mobile pages for my blog yet, but that is what I plan to do next.

WIP: Resource Preview

2011-06-03T09:00:35+00:00

This is the fourth in my series of Web Integration Patterns. Check out the intro at this URL http://rollerweblogger.org/roller/entry/web_integration_patterns

Synopsis

Enhance links shown in HTML pages so that users can hover, mouse-over, or use some other gesture, to view a preview of the resource at the other end of the link.

Motivations

Make it convenient for a user to get information about a link but without having to navigate to the link and without having to leave the current web page in the browser.
Make applications appear to be part of one integrated whole by enabling them to delegate to each other's user interfaces for preview display.

Complements

Links
Delegated Resource Creation & Selection
Common Navigation

This pattern allows users to view a preview of links that are displayed in a web page. This preview might be a simple tool-tip with plain-text, a more rich JavaScript-driven display with hyper-text and graphics, or it might be a thumbnail view of the page at the other end of the link.

Resource Preview works well for web applications that use the Links pattern for integration and use links to establish relationships between resources. If a user is looking at a Bug Report, for example, they might see a list of links to associated resources like attachments to the bug, test cases impacted by the bug, etc. The user can hover over links and see a preview with key information for each link, without leaving the current web page that they are on. Below are three different approaches to applying the Resource Preview pattern in web applications.

Approach #1: Preview Site offers previews to other applications

With this approach, there is a special preview site that crawls a set of web sites and creates Resource Previews of the web pages of those sites. Then, web sites that wish to display previews on links use some special JavaScript magic to display those previews. An example of this preview site approach is Snap.com’s Snap-Shots service. Anybody with a web site can sign up for the service, get some special HTML code that can be added to their site to display a graphical and thumbnail-style preview whenever a user hovers over a link.

Here’s an example of a Snap.com preview on the company’s blog:

The advantages of the preview site approach are participating sites don’t have to do anything except for adding a small bit of JavaScript code to enable previews. The disadvantage is that the automatically generated previews often can’t offer more than a page title, thumbnail image and perhaps a short excerpt from the page being previewed. Also, it’s interesting to note that there was a backlash against Snap-Shots and some people thought they were distracting and gimmicky. Regardless of that criticism, this approach to Resource Previews is not very interesting from a Web Integration Patterns point of view because the focus is making one site more useful and interactive, and not integrating with others.

Approach #2: Application creates and displays previews as needed

With this approach, an application that needs to provide preview creates the previews that it needs and take case of displaying them. The best known examples of this approach are probably the preview features built into Microsoft’s Bing and Google’s search engine. This approach is also not very interesting as a WIP because its all about making one site more useful, not integrating with others.

Here’s an example of a Resource Preview in Microsoft Bing:

Approach #3: Applications offer previews of their own resources

This approach is to enable web applications provide previews of their own resources and make them available via a simple protocol. The best example of this approach is OSLC UI Preview, which is part of the OSLC specifications. The OSLC specs are designed to enable integration between ALM tools, but can be applied to integrating web applications of almost any stripe. Applications that implement OSLC UI Preview provide a UI Preview for each resource, one that provides a summary of the resource and links to large and small previews of the resource.

Here’s an example of an OSLC UI Preview from FusionForge, an open source ALM suite:

Here’s an example of a Resource Preview from IBM Rational Team Concert, which also implements the OSLC UI Preview spec:

There are several advantages to this approach. Preview-providing applications have complete control over the content and appearance of previews for their resources. Preview-consuming applications don’t have to create and format them, they just display what comes back from the provider. No special preview site is needed to enable previews. Another advantage to this approach to Resource Previews is the OSLC UI Preview specification, which defines an open and standard way to provide and consume previews.

Wrapping up

I've explained what I call the Resource Pattern and three different ways that I've seen it used. Resource Preview is a good way to add value to the Links pattern and to make links even more useful. The best way to implement this pattern is to ask all web applications that want to integrate to offer Resource Previews via a standard protocol such as OSLC UI Preview, and to display them whenever a user hovers over a link.

OSLC Core v2 specification now FINAL

2011-06-01T14:38:13+00:00

I've been working on the OSLC Core specification for about 1.5 years now as workgroup lead, and OSLC fits squarely under the "open web technologies" and Web Integration Patterns topics of this blog, so I'm blogging this happy news.

Here's the announcement From the OSLC Core Workgroup mailing list:

From: Dave Johnson
To: oslc-core (a) open-services.net, community (a) open-services.net
Subject: OSLC Core v2 specification now FINAL

Today [1], I'm very happy to announce that the OSLC Core v2
specification is FINAL.

The OSLC Core v2 specification [2] defines a set of REST and Linked
Data-based patterns, resources and protocols for integration of application 
and product lifecycle resources (ALM and PLM). It's designed to be the
foundation for all other OSLC domain specifications and there are now
three final OSLC specifications that are based on the Core, those
being the OSLC Change Management (CM) [3], OSLC Quality Management
(QM) [4] and OSLC Requirements Management (RM) [5] specs.

I'd like to thank all of the members of the OSLC Core Workgroup and
community for their hard work, critical thinking and ability to work
together in such a productive and pleasant way. Also, special thanks
to those OSLC domain workgroups who rebased their work on the Core and
development teams that provided excellent feedback along the way.

Thanks,
- Dave

--
David M. Johnson
OSLC Core Workgroup Lead
IBM Rational Software

[1] Move to final was proposed last week, along with a small set of
changes which have since been applied to the specification. 
[2] OslcCoreSpecification
[3] CmSpecificationV2
[4] QmSpecificationV2
[5] RmSpecificationV2

I really do have another Web Integration Patterns post on the way shortly, so stay tuned.

Apache Roller 5.0 released

2011-05-25T14:06:00+00:00

(cross-posted from the Roller project blog)

Here's some more happy Roller news. Apache Roller 5.0 has been released!

http://rollerweblogger.org/project/mediaresource/3cdaff7b-2745-4dac-89c9-151a3a1ccf26' align='right' style='padding:1em' />

The major new feature in Roller 5.0 is Media Blogging, a set of enhancements to Roller's file upload and management capabilities. Also included in 5.0 are simple multi-site support, OpenID and ~OAuth support for Roller's AtomPub interface. All major dependencies have been updated and Roller now uses Maven for build and dependency management. You can find a summary of Roller 5.0's new features on the Roller wiki.

The road to Roller 5.0 has been a long one and if you are interested the history, you might want to check Dave Johnson's What's New in Roller 5.0 presentation from ApacheCon US 2009. Roller 5.0 includes contributions from contributors from Google Summer of Code, San Jose State Univ. and the usual case of Roller committers. Thanks to all who contributed to Roller 5.0 over the years.

To download Apache Roller 5.0 and documentation, visit the Apache Roller download page at the Apache Software Foundation's website.

Welcome new Apache Roller committer Shelan Perera

2011-05-23T17:53:33+00:00

(cross-posted from the Roller project blog)

Here's some happy news. A new committer has joined the Apache Roller project. Shelan Perera has been helping out on the mailing lists, submitting fixes and recently won a Google Summer of Code (GSOC) project to add mobile blogging features to Roller. He was nominated for committer-ship and voted in on May 5, 2011.

Shelan's GSOC project is to add mobile theming capabilities to Roller. You can find the Mobile Theming for Roller proposal on the GSOC website. Shelan is seeking feedback on requirements and design for the project, and keeping the community in the loop by running a blog to journal his progress: Apache Roller Mobile Platform.

Welcome Shelan!

Inside Out Half-Marathon 2011

2011-05-22T16:47:27+00:00

I'll be resuming my Web Integration Patterns blog series this week, now that I have some "free" time again. It's been a busy Spring at work and at home, especially during April and May. My early mornings, evenings and weekends were consumed by half-marathon training, (and the Apache Roller 5.0 release, more about that later).

As for the half-marathon, it went very well. I really enjoyed training and running with my oldest son. The race was today and we're both pretty happy with our results, which you can see below.

13th Annual Inside-Out Sports Classic - Half-Marathon

Place Bib  Name                    S Ag City             St Chiptim Guntime Pace  
===== ==== ======================= = == ================ == ======= ======= ===== 
  247  308 ALEX JOHNSON            M 14 RALEIGH          NC 1:59:10 1:59:18  9:06
  154  309 DAVE JOHNSON            M 47 RALEIGH          NC 1:50:34 1:50:43  8:27

The race course is wonderful and runs through Umstead park on the small-grain gravel bridle trails there. Here's the route map I made on dailymile.com (check out the Elevation Profile, it's a hilly one).

WIP: Embedded Properties in HTML

2011-04-07T21:57:52+00:00

This is the third in my series of Web Integration Patterns. Check out the intro at this URL http://rollerweblogger.org/roller/entry/web_integration_patterns

Synopsis

Enable easier integration and better search across integrated web applications and sites by using standard mechanisms (e.g. Microformats, RDFa) to embed property values in HTML pages.

Motivations

Enable integrated web applications to read, parse and make use of each others data
Enable better search across integrated web applications

Related Patterns

Links
Web APIs
Linked Data + REST

In this pattern, you define different types of resources or “objects," and the properties may be associated with each, or you adopt an existing vocabulary. You define a way to embed type information and properties values into HTML, by embedding name and value pairs in a way invisible to a user or by marking up strings of visible text as property values, or you adopt an existing technique.

For example, a cooking web site could share recipe data using this pattern by presenting each recipe with embedded property values to make it easy for other web applications, mash-ups and search engines to parse out the different parts of the recipe, title, ingredients list, list of steps, etc. This is a real example, by the way, Google indexes recipe data in HTML pages marked up in either Microformat-style or RDFa.

If you present each web page as a resource with a type, or multiple types, and embedded property values then you don’t need to define and provide a special “REST API” to allow other web applications to access your data.

This pattern can enable more powerful search capabilities across integrated web application because search engine’s can be aware of and provide special indexing for those different types of resources and properties.

There are at least three different standard ways to embed data in HTML: Microformats, HTML Microdata and RDFa, each of which I’ll explain below.

Microformats

According to Wikipedia “Microformats emerged as part of a grassroots movement to make recognizable data items (such as events, contact details or geographical locations) capable of automated processing by software, as well as directly readable by end-users.” A community grew around Microformats and this community has worked to define Microformats for a variety of use cases. The microformats.org web site lists nine microformats, including hCard for sharing contact information, hCalendar for sharing events and hReview for sharing movie and book reviews.

The basic idea of Microformats is to use existing HTML element attributes, specifically rel and class, to convey types of things and properties of those things. Here’s an example from Wikipedia: to represent information about a geographic location in HTML, the Geo Microformat uses the syntax below:


   The birds roosted at
     <span class="geo">
     <span class="latitude">52.48</span>,
     <span class="longitude">-1.89</span>
   </span>

The outer-most span’s class element indicates the type "geo" and the inner spans carry the "latitude" and "longitude" property values.

If you want to take advantage of Microformats in web application integration, then you pick one of the existing formats or you invent a new one, ideally by working with the Microformats community to do so.

Another approach for implementing Embedded Properties in HTML is HTML itself.

HTML(5) Microdata

HTML Microdata is a W3C working draft, a specification that defines ways to embed property values in HTML. As Mark Pilgrim explains in Dive Into HTML5, HTML Microdata “annotates the DOM with scoped name/value pairs from custom vocabularies.” This is very similar to what Microformats do, but instead of “overloading” the class attribute, HTML Microdata adds some new HTML attributes to enable embedded property values.

HTML Microdata enables you to mark-up an HTML element as an item that contains properties. You do this via the itemscope attribute, you indicate the item’s type via the itemtype attribute and each property is indicated by the itemprop attribute. Here’s an example, also from Pilgrim’s book:


   <TABLE itemscope itemtype="http://data-vocabulary.org/Person">
     <TR><TD>Name<TD>Mark Pilgrim
     <TR><TD>Link<TD>
       <span itemprop="url">
       <A href=# onclick=goExternalLink()>http://diveintomark.org/</A>
       </span>
   </TABLE>

The table element defines the scope of the item and the type of the item is a URL http://data-vocabulary.org/Person which indicates that the item represents a person. The enclosed span element carries the “url” property of the person.

It’s good to have a standard way to encode types, scope and property values in HTML, but you also need standard vocabularies of types and properties to make this approach successful. There are some HTML Microdata vocabularies defined at data-vocabulary.org and HTML Microdata can also be used with RDF, opening up a huge number of vocabularies.

RDFa

RDFa is a W3C recommendation, a specification that defines ways to embed RDF data in HTML and XHTML. Like HTML Microdata, RDFa adds new attributes to HTML to enable this. Let’s look at an example from the Wikipedia page on RDFa, which uses two of these attributes about and property:

 
   <div xmlns:dc="http://purl.org/dc/elements/1.1/"
     about="http://www.example.com/books/wikinomics">
     <span property="dc:title">Wikinomics</span>
     <span property="dc:creator">Don Tapscott</span>
     <span property="dc:date">2006-10-01</span>
   </div>

The above example embeds some property values about the book Wikinomics, which is identified by URL http://www.example.com/books/wikinomics. The example provides a title, creator and date values about that resource, and it doesn’t say what type of resource exists at the URL, but it could.

The advantage of the RDFa approach is that it ties in with the growing RDF / Linked Data movement, and the huge number of vocabularies from almost every field of endeavor. RDFa has some momentum and has been adopted by Facebook in its OpenGraph Protocol and Best Buy for exposing catalog data in standard ways. However, HTML Microdata can also be used with RDF, so RDFa is not the only way to go if you favor Linked Data.

Wrapping up

We've talked about three different ways to implement Embedded Properties in HTML, so which one do you choose? It's not to me whether one of these techniques will dominate. Google is hedging its bet, so to speak, by indexing all three types of data. If you don't already have a favorite, one way to choose is to look at what others in your community, company or "industry vertical" are doing. The network effect applies here, so if one technique is already favored amongst web application or sites in your space, then go with that.

Next up: Resource Preview

WIP: Common Navigation

2011-03-27T10:08:44+00:00

This is the second in my series of Web Integration Patterns. Check out the intro at this URL http://rollerweblogger.org/roller/entry/web_integration_patterns

Synopsis

Make separate web sites and applications appear to be one by using common user interface elements for navigation.

Motivations

Provide easy navigation between integrated web sites & applications.
Make separate web sites appear to be one and parts of the same overall user interface.

Related patterns

Links

This pattern is an extension of the Links pattern. The idea is to use links combined with common user interface (UI) elements, as a way to provide navigation between integrated sites and to make the separate sites appear to be parts of a whole application. This pattern is usually implemented via a banner with links, a tabs or some other type of menu component.

To make this work, somebody has to create the common navigation, give it an attractive design, decide which web sites or applications are included in the navigation. Consequently, this pattern works well for a set of web sites owned by the same organization, or a suite of packaged web applications.

Pros and Cons

The advantages of Common Navigation are that it works, it does make separate web sites seem to be part of one integrated whole and it's relatively easy to implement. One disadvantage of this approach is that the Common Navigation elements can either conflict with, in a visual sense, or distract from the web sites themselves. But if you're selling a suite of web applications, you've got control of whole design and you can solve this problem.

Examples

Here are some examples of Common Navigation that I see almost everyday. The first is Google.

Common Navigation in Google web applications

Google Mail, Calendar and other Google apps all share a Common Navigation bar across the top of the page. Google decides which apps appear in the menu, and the tech blogs whine whenever an item is moved or removed from the line-up. This is a pretty shallow integration: just links to applications and no more.

Now, let's see a deeper example.

Common Navigation in IBM Lotus Connections

IBM Lotus Connections is a suite of social software applications including social networking, blogs, wiki, forums, file sharing and etc. As you can see below, Common Navigation is used to provide links, not just to applications, but into specific parts of applications. For example, the Latest Entries page in the Blogs app, and the Wikis that I own vs. Public Wikis.

Next, another suite example.

Common Navigation in IBM Rational Team Concert

IBM Rational's Jazz-based products use Common Navigation, but with a project-orientation, showing a user's project's within each application. You can see this below in Rational Team Concert. A development project spans different applications and may have requirements managed by one web application, defects tracked by another, test cases managed by a third and so on. Each user sees the right menu for their projects.

Next, an example that's not a suite of web applications.

Common Navigation in StumleUpon

Another example is StumbleUpon. When you use StumbleUpon, you see the banner at the top of every page. You press the Stumble! button and a you see a randomly selected web page from anywhere on the web, or from some specific category of web site, and you still see the StumbleUpon banner. The StumbleUpon banner makes the whole web seem like one big site, deigned for "stumbling" around and sharing the things you find with friends.

Wrapping up

I wasn't sure that Common Navigation really deserved its own pattern, as it's really just a simple and obvious application of the Links pattern, but I think this works. As always, feedback is welcome. What did I get wrong? What did I leave out?

Next up: Web Annotations.

Web Integration Patterns

2011-03-15T09:30:40+00:00

In my previous post, I promised to write a blog series on Web Integration Patterns. This post explains the concept and a bit about how I plan to write about it.

What I’m calling Web Integration Patterns are techniques for integrating software systems, web applications and web sites using the common technologies of the web. These patterns build on HTTP, HTML, JavaScript and sometimes JSON, RDF and XML to provide ways to integrate software systems and include both programmatic approaches and user interface integrations.

Not the same as Enterprise Integration Patterns

These patterns are different from Enterprise Integration Patterns (from the book of the same title). Those enterprise patterns make the most sense inside an organization where shared-databases and specifically messaging middle-wares are part of the common infrastructure. Web Integration Patterns assume only the web as common infrastructure, or at least the best of them do, and that means they can work equally as well in a controlled enterprise Intranet as they do on the big wild Internet. They can work well for a suite of web applications or a family of loosely related web sites.

Four categories of patterns covered

I plan to write about a dozen blog entries each covering one pattern, and I’ll do at least one per week. I’ll start with the basic patterns then introduce more advanced patterns. I’ve organized the patterns into four groupings, listed below in the order that I will cover them:

(I'll add links in below as I publish each pattern)

Basic Patterns: fundamental patterns which are relatively easy to implement, and, mostly, possible on a website made up of static HTML pages.
- #1 Links: Use links as a way to integrate web sites & applications via navigation and relationships between resources.
- #2 Common Navigation: Make separate web sites and applications appear to be one by using common user interface elements for navigation.
- #3 Embedded Properties in HTML: Enable easier integration and better search across integrated web applications and sites by using standard mechanisms (e.g. Microformats, RDFa) to embed property values in HTML pages.
- #4 Resource Preview: Enhance links shown in HTML pages so that users can hover, mouse-over, or use some other gesture, to view a preview of the resource at the other end of the link.
- #5 Widgets and Gadgets: Allow other web sites and applications to integrate your site into their web pages by providing an embeddable user interface, commonly known as a Gadget or Widget
- #6 Feed-based Integration integrating web sites and applications by using standard feed formats, e.g. RSS and Atom, to convey timely information, updates, status messages, events and other information
Advanced Patterns: more advanced patterns, which may require some server-side logic and infrastructure beyond a plain old web server.
Authentication Patterns: these are advanced patterns that concern authentication and authorization of users and web sites, including Single Sign On, OpenID and more.
Social Patterns: patterns from the world of social networking, including things like OpenSocial, Facebook apps and ActivityStreams.

Perhaps using the word “patterns” is a bit pretentious. I’m no gang of four or even one, and I don’t plan to give these patterns the comprehensive treatment examples that you’d find in a patterns book. I won’t be providing source code examples either. Still, I think Web Integration Patterns is the right name and format, so I’m sticking with it.

The first pattern that I’ll cover is Links. More Later...

More than bloggy

2011-03-07T06:00:00+00:00

I’ve been meaning to write on this topic for a some time and to explain how I’ve gone from being an advocate of RSS/Atom feeds, Atom Publishing Protocol and things bloggy to being a proponent of Linked Data (video), Semantic Web, RDF and other things that I previously considered to be nuisances. I've also got a new topic and blog series to announce, so here goes.

Once upon a time, I actually said “the web is bloggy” in conference presentations and without shame. It sounds silly and, for a blog server developer self-serving, but it’s true and it’s still true to this day. So much of what we do on the web can be made more useful via feeds. That’s because so many things can be usefully represented as a well formed log entry, i.e. a URL, title, date, description and a chunk of content. No matter whether your web application is Flickr, Twitter or IBM/Rational Team Concert, you can enable all sorts of interesting integrations and mashups by providing feeds.

Atom Publishing Protocol (APP) applied the feed concept to web APIs, using a feed to represent a collection of entries and allowing API callers to use HTTP to create, retrieve, update and delete entries. Entries could be anything of course. In the case of Google’s APP-based GData Protocol, entries are Calendar Events, Picasa Photo uploads or YouTube videos. In the case of the IBM/Lotus Connections API, entries are Forum Posts, Social Network Profiles, File Uploads and more.

Feeds only get you so far

I’m not going to argue that RSS is dead (or Atom) because I don't believe that, but the feeds approach only gets you so far. With feeds, we agree on a small set of common properties, e.g. URL, title, date, description and content. Not all resources on the web have those properties and many resources have a lot more. So in some cases applying Atom means forcing things to fit into the Atom model and in some cases it means inventing new properties for Atom, something that is supported by Atom format.

APIs only get you so far

I’m not going to argue that APP is dead or a failure either, but APP is an API approach and, for web integration, APIs only get you so far. It’s definitely a good thing for developers when a web site or application provides an API, but if you have to integrate a set of N web sites and each has it’s own API and (even if they’re all based on APP) you’re stuck doing N x N point-to-point integrations. APIs are not necessarily the best way, and certainly not the only way, to enable integrations on the web.

Linked Data

Linked Data (W3C page) is a different approach. Instead of forcing everything to fit the blog model, Linked Data gives us a way to define common vocabularies of properties about resources on the web and, most importantly, the links between them. Linked Data enables integration by allowing sites to weave their resources into the interlinked web of data that is the web.

Next: Web Integration Patterns

These days, what's important to me at work is integration and in my mind, Feeds, APIs and Linked Data are different “patterns” for web-based integrations. They can be used in complementary ways. Just as we have defined and cataloged Enterprise Integration Patterns, I think it will be useful to do the same for Web Integration Patterns. I know I'll learn something in the process. In my next posts, I'll expand on this concept and after that, I'll start cataloging, in an informal and bloggy way, the patterns that I'm familiar with. More later...

Roller 5 and JBoss 6

2011-01-20T18:00:15+00:00

In my quest to make Roller work on Java EE 6, the next server that I tackled was JBoss 6. In this blog I'll describe how I approached the problem what I learned along the way.

Tested with Hibernate JPA

Roller uses JPA for database storage and specifically the Apache OpenJPA implementation. I knew that JBoss uses the Hibernate JPA implementation and I suspected that there would be JPA portability problems, so I decided to run Roller's JUnit tests against Hibernate JPA. There were many test failures and fortunately, the failures were easy to fix.

Where OpenJPA is lenient, Hibernate JPA is strict about declaring transients fields as transient. So, wherever Hibernate complained, I added the appropriate transient declaration and soon all tests were passing. There were a lot of changes, but they were all trivially easy. Since I first "ported" Roller to ElipseLink JPA, its possible that some of the changes I made for EclipseLink helped with the port to Hibernate JPA.

Those of you who are familiar with Roller's history might remember that this is the second time I worked to make Roller work with Hibernate. Early versions of Roller ran on Hibernate until Roller 4, when we ripped it out because Apache policy does not allow LGPL. With Java EE 6, Roller can run on Hibernate and we don't have to ship the Hibernate jars to do so.

Tried using the Java EE version of the Roller WAR

Since JBoss 6 is a Java EE 6 server, just like Glassfish, I figured I could use the same WAR that I created for Glassfish. That didn't really work out, as you will see below. When I attempted to deploy the Roller WAR to JBoss I ran into two problems:

Problem 1: Xerces and JavaAssist

When I tried to deploy the Roller WAR to JBoss, I ran into class-cast exceptions that indicated that the version the the Xerces XML parser included with Roller conflicts with the one that is included in JBoss. I encountered a similar problem for the JavAssist jars, which are also part of Roller. This was quite surprising to me. Apparently, JBoss uses Xerces and JavAssist internally and for some reason the JBoss internals bleed through and interfere with applications; seems like a bug to me. So, we have to have a special Roller WAR for JBoss without the Xerces and JavaAssist jars in the Roller WAR.

Problem 2: JNDI Datasource Name

The next problem that I encountered was the datasource name. Roller uses the JNDI naming API to lookup its JDBC datasource. In all of the other app servers, we tell people to setup a datasource with the JNDI name 'jdbc/rollerdb' but that name did not work for JBoss. For JBoss, I could only get names of names of the format "java:/name" to work. Unfortunately, with JPA the datasource name must be embedded in the persistence.xml file which is embedded in a JAR file which is embedded in the Roller WAR file. It's in there deep, so we have to produce a special Roller WAR for JBoss with a JBoss-friendly datasource name.

*Please* correct me if I'm wrong. I would love to be wrong about either of these two problems.

Created special Roller WAR just for JBoss

Due to those two problems, I modified the Roller build process to create a special Roller WAR for JBoss without the OpenJPA, Xerces and JavAssist JARs and with the JBoss friendly JNDI name java:/RollerDS inside all included persistence.xml files.

Deployed, tested and updated the docs

Once I worked around those two problems, installing Roller on JBoss was easy. I did the whole thing via the JBoss web console, which was not familiar to me but was pretty easy to understand and use. I documented the whole process in the Roller 5 Install Guide (2MB PDF), with screenshots.

Next up: Roller 5 on WebSphere 8 (beta)

Roller 5 and Glassfish 3

2011-01-10T17:29:36+00:00

In my quest to make Roller work on Java EE 6, the first server that I decided to tackle was Glassfish 3. In this blog I'll describe how I approached the problem and what I learned along the way.

Tested with EclipseLink JPA

Roller uses JPA for persistence and specifically the Apache OpenJPA implementation. I knew that GlassFish uses the EclipseLink JPA implementation and I suspected that there would be JPA portability problems, so I decided to run Roller's JUnit tests against EclipseLink JPA. I wanted to find and fix those problems before even touching GlassFish. The tests ran and there were many JPA related failures and errors, most due to differences in the way that EclipseLink handles bi-directional relationships and the use of unmanaged objects.

Where OpenJPA is lenient about bi-directional relationships, EclipseLink requires you to manage both ends. For example, with OpenJPA you can get away with adding a bookmark to a folder with folder.addBookmark(bookmark), but with EclipseLink you'd also have to add bookmark.setFolder(folder).

Where OpenJPA is lenient about use of unmanaged objects, EclipseLink will complain bitterly whenever it finds one in a persisted collection or relationship. This was more of a problem in the tests than in the actual Roller code, which usually deals only with managed objects (i.e. those persisted to / loaded from the database and managed by the JPA implementation).

I was able to fix those problems easily and move onto the next step.

Created a Roller WAR without OpenJPA

I also knew that Roller included some jars needed for Tomcat but that are not needed, or worse, problematic, on true Java EE servers. For example, we include the Apache OpenJPA implementation because Tomcat doesn't have one of its own. I modified the Roller build process to create two builds. One build is for Tomcat and includes OpenJPA and the other build is for Java EE and does not.

Deployed, tested and updated the docs

Next, I figured out how to deploy Roller to GlassFish 3 via the GlassFish web console, which was pleasant to use and familiar to me. I didn't run into any problems along the way and later I documented the whole process in the Roller 5 Install Guide (2MB PDF), with screenshots.

Next up: Roller 5 on JBoss 6

Roller 5 and Java EE 6

2011-01-05T08:15:53+00:00

It's hard to believe, but I've been dorking around with Roller, the blog software that powers this site, for almost 10 years now. I started in summer 2001. In the past couple of years, I've had a lot less time to work on Roller. I devoted some of that time to mentoring student developers, which was fun and rewarding. I also spent time making Roller more consumable for developers by making it easier to build, run and deploy to modern Java app servers, which was not really fun but was definitely educational, bloggable even.

Roller 5 as Java EE 6 case study

Making Roller work on the new crop of "modern" Java EE 6 servers was an interesting experience and the story makes a pretty good case study in Java EE application portability. I've put together a short series of blog posts to tell the story and this is the first, an overview. In subsequent posts I'll explain the changes I had to make to get Roller working on:

	GlassFish 3 (released June 2010)
	JBoss 6 (released December 2010)
	WebSphere 8 (beta released July 2010)
	Tomcat 6 and 7 (not Java EE but Roller just has to work on Tomcat)

There are other Java EE 6 servers out there and there's a good summary of them on The Aquarium; those are just the ones I had time to explore.

Overview of Java EE 6 changes

There were three categories of things I changed to accomodate Java EE 6 servers. The first is changes to the Roller code base to make Roller work on all of my targets. Most of these changes were in way Roller uses calls the Java Persistence APIs (JPA), needed because of differences in the JPA implementations used by each server. GlassFish uses EclipseLink JPA and JBoss uses Hibernate JPA. On WebSphere and Tomcat, Apache OpenJPA is the implementation. Fortunately, nowhere did I have to use any conditional code or introduce special behavior for any platform.

The second category of changes is additions to the Roller install guide to mention the special settings required to make Roller work on all targets. On some platforms, special behaviors must be enabled for Roller, for example the "filter compatibility" flag on WebSphere. I added a section to the install guide for each server and documented the details there.

The third category is changes to the Roller build process to create special versions of Roller for some servers. Unfortunately, due to Tomcat not being a full Java EE server and some JBoss classloader issues, I'm not able to provide one Roller release that runs on all servers. The latest Roller 5 release candidate comes in three flavors, one for Tomcat 6/7, one for JBoss 6 and one for "Java EE" which is the one intended for use with Glassfish 3.1 and WebSphere 8.

Up next: GlassFish 3

In the next post I'll tell you what I had to do to get Roller running on Glassfish 3 with EclipseLink JPA.

Apache Roller 5.0 RC3

2011-01-04T08:00:00+00:00

On the last day of 2010, I made available the third release candidate for Apache Roller 5.0. The main difference between this new candidate and the previous one is that the new RC3 runs on Java EE 6 servers: Glassfish 3, JBoss 6 and Websphere 8 (currently in beta). Making this happen took a lot more work than I expected and I'll blog about that over the next couple of weeks as it is an interesting case study in Java EE 6 portability.

Here's the announcement:

Apache Roller 5.0 Release Candidate RC3 is now available for testing.
Note that this is NOT a release of the Apache Software Foundation or
anybody else; this release candidate is for testing purposes only and
not recommended for production.

   What's new in Roller 5.0:
   https://cwiki.apache.org/confluence/display/ROLLER/What's+new+in+Roller+5.0

   Change list (issues resolved since 4.0)
   http://bit.ly/gAhDWR

   Issues resolved since last release candidate (RC3)
   http://bit.ly/dZ27Nx

   Signed binary and source files. Also, documentation in PDF form
   http://people.apache.org/~snoopdave/apache-roller-5.0/

The biggest change in RC3 is the new support for Java EE 6 application
servers: Glassfish 3, JBoss 6 and Websphere 8 (beta). I've been able
to verify that Roller runs on all of those servers, and I updated the
installation guide to explain in detail how you install on Glassfish,
JBoss and WebSphere.

If you would like to help out then please test RC3, discuss the
problems you encounter here and file specific bugs with steps to
reproduce in the Roller JIRA bug tracking system.

Thanks,
Dave

That announcement is available here: http://markmail.org/message/my5wbld2xqvhqpyg

Apache Roller 5.0 RC2

2010-10-03T10:11:39+00:00

A couple of weeks ago, I made a second release candidate available for Apache Roller 5.0. Here's the announcement (also available at http://s.apache.org/apacheroller50rc2):

Apache Roller 5.0 Release Candidate RC2 is now available for testing. 
Note that this is NOT a release of the Apache Software Foundation or 
anybody else; this release candidate is for testing purposes only and 
not recommended for production.

   What's new in Roller 5.0:
   https://cwiki.apache.org/confluence/display/ROLLER/What's+new+in+Roller+5.0

   Roller 5.0 JIRA change list:
   https://issues.apache.org/jira/sec ... sion=12313828

   Signed binary and source files
   http://people.apache.org/~snoopdave/apache-roller-5.0/

   Issues resolved since RC1:
   http://bit.ly/9eWjJk

If you would like to help out then please test RC2, discuss 
the problems you encounter here and file specific bugs with steps to
reproduce in the Roller JIRA bug tracking system.

I'm running RC2 on this site and it seems to be holding up just fine so far.

Forces and vulnerabilities of the Apache model

2010-05-21T13:05:30+00:00

Ceki Gülcü: Instead of trying to learn from past failures which open discussion is supposed to encourage, Apache forges on in the path of egalitarianism. As time passes, I see attempts at institutionalizing egalitarianism instead of recognizing its inherent injustice. If egalitarianism is really at the core of the Apache way as an absolute value, then the Apache way sucks. Yay!

While the one person one vote principle applies to a democracy in order to run a country for the benefit of all, the one person one vote principle is ill-suited in a purported meritocracy the size of Apache. If it must be "one developer one vote", then the word meritocracy cannot be honestly ascribed to Apache.

Very interesting discussion and comments on the dynamics of meritocracy at the Apache Software Foundation from Ceki Gülcü.

New era for JSPWiki

2008-11-26T08:46:20+00:00

Janne Jalkanen: We are now beginning a whole new era with becoming an Apache project and JSPWiki v3, which will signal the first major overhaul of the entire software since v2.0 in 2002. We've got a bunch of good committers (with a new one added this weekend - welcome, Florian!) and a bunch of pretty exciting things we want to do.

I'm so happy to see JSPWiki thrive at Apache, even in the incubator. You may remember that I wrote to Janne back in Summer 2007 and suggested the move, and I've crowed about it before, but Craig Russell is the one who stepped up to mentor the project and, from what I've seen, he's doing a great job.

Atom news: Apache Abdera graduates

2008-11-21T15:54:48+00:00

Congratulations to the Apache Abdera team, who've just graduated to full Apache top level project status. The don't have the new site at abdera.apache.org up yet and they're still not quite at 1.0 yet, but this is a major milestone. They've got the best Atom format and protocol toolkit around, in my opinion.

via Garett and James.

Roller status

2008-08-18T18:27:40+00:00

If you want the lowdown on what's going on with Roller community health, ongoing work and upcoming releases then check out the Apache Roller August 2008 Board Report.

Six

2008-03-08T17:30:36+00:00

By my count that's how many Apache members work at Sun. I thought I had a complete count, but Nick Kew's recent post revealed a sixth (see the comments). Here's the list:

Craig McClanahan
Craig Russell
Dave Johnson
Jim Winstead
Nick Kew
Ted Leung

Know of any other Sun employees that are Apache members?

Welcome to Sun!

2008-03-04T18:33:38+00:00

It's great to be welcoming new folks to Sun, especially when they're brilliant people like Ted Leung and Nick Kew, both of whom, by the way, are members of the Apache Software Foundation. I met Ted at ApacheCon US 2004 in Vegas and he answered all my questions about the implications of moving Roller to the ASF. And I met Nick at ApacheCon EU 2006 in Dublin and we chatted, over a couple of pints of Guiness, about the perils and pleasures of working from home and other things.

I'm also pretty damn pleased to be part of the MySQL welcoming committee, AKA the SunVisor program, and paired-up with Chuck Bell of MySQL. He's the author of Expert MySQL. I'll be answering his questions about Sun and, I hope, learning a thing or two about MySQL in the process.

Welcome to Sun guys!

SpringSource and Covalent: good thing for Apache?

2008-01-30T09:27:34+00:00

SpringSource, the company behind the Spring Framework, has purchased Covalent, a company that provides support for Apache projects. This popped up on my radar because Covalent offers support contracts for Roller and in fact, SpringSource CEO Rod Johnson mentioned Roller specifically when talking about the deal (emphasis mine):

<a href= "http://www.informationweek.com/news/showArticle.jhtml?articleID=205920751">Rod Johnson: "We want to support the open source software that people want to use," including the Geronimo application server, the Axis Web Services Framework from Apache, and the Apache Roller Blog multi-user blogging software."

Sounds like a good thing and hopefully it will improve the support story for all Apache products. In fact, it could be a really good thing for Apache projects because Rod's philosophy is that you can't support software unless you are one of the software's creators.

<a href= "http://www.infoq.com/news/2007/06/open-source-models">Rod Johnson: "You can't divorce the process of maintaining software from the process of creating software...That's not the future of enterprise open source - unless open source has no future"

Based on that, we can assume that SpringSource will now be paying committers to do creative work on Roller and other Apache projects so that they can provide the best maintenance and support of those same projects. Right? Maybe I'm too naive -- after all, I figured having Roller in Lotus Connections meant IBM would be contributing.

Apache Shindig voting in progress and more OpenSocial details emerge

2007-11-30T12:48:25+00:00

I wrote about Shindig before, it's a new open source project to implement the Google OpenSocial APIs. Well, now the official voting to accept the Shindig project into the Apache Incubator is in progress and some interesting details have emerged in the latest version of the proposal. First, as you can see by the initial list of committers in the proposal Google has joined the Shindig effort in force. Second, the proposal says that Shindig will be the reference implementation of the OpenSocial APIs. And third, Shindig will not only include the client-side JavaScript container but also a Java back-end. Brian McAllister has already made some "gnarly" initial client-side container code available, I can't wait to see the Google contribution.

ApacheCon US 2007 wrapup

2007-11-20T17:42:51+00:00

I spent last week at the ApacheCon US 2007 conference in Atlanta, Georgia. Here's a write up of some of my experiences there.

The Hackathon

The Hackathon is a two day event. For those who don't know what a hackathon is, I'll explain how it works. A room with about 20 large round tables, wireless internet and plenty of power strips is provided. Folks show up, plug-in, start working and chatting with each other about projects and other random topics. Sometimes they cluster by project, Struts guys around one table and Geronimo guys at the next, but usually there's a good mix at each table. Sometimes groups have specific goals like "ship version 2.0" in mind, but usually folks are just working, sharing knowledge and getting to know each other. At various times pizza, soda, breakfast pastries, beer, coffee and/or wine is carted into the room -- so there's not much reason to leave except for things like fresh air, sunlight and sanity.

I had a couple of interesting conversations at the Hackathon. I spent some time talking to Brian McAllister about his Shindig proposal, which I've already mentioned. I also spoke with David Jencks about getting better Role Based Access Control (RBAC) in Java EE to support social software like blogs, wikis and social network applications because static roles hard-coded into web.xml just won't do.

Roller and the Roller/Struts2 BOF

As usual, I attended ApacheCon to promote and represent Roller and I did so in my talk, at the Roller/Struts 2 BOF, at the Sun booth and wherever else I could. As I mentioned before, my talk on Roller and blogs as a web development platform went well. I had updated my talk to include an example of how to post to a blog using Abdera and I urged folks to stick around for Garrett Rooney's Abdera talk, which followed mine in the same room.

The BOF session also went well. Matt Raible and Don Brown organized the session and all I had to do was show up and enjoy the good company and the Atlassian provided beer, wine and nibbles. About ten people showed up and we talked about a variety of topics including user management in Roller, the growing similarity of Struts 2, Spring MVC and other Java web frameworks and then my wife dragged me away from the fun. By the way, Don works for Atlassian on the new JIRA Studio suite, which, I believe, is going to give Collabnet, SourceForge Enterprise and even the likes of IBM Jazz and Microsoft Team System some serious competition.

Roller the Java Content Repository (JCR) API

The idea of using a content management system to store Roller content keeps on coming up. At ApacheCon EU earlier this year, I spent some time talking to Lars Trieloff (who now works for CMS vendor Day Software) about implementing the Roller back-end interfaces using the Java Content Repository (JCR) APIs instead of the Java Persistence API (JPA) that we use now. At this ApacheCon, Noel Bergman brought up the topic a couple of times and pointed out that Day Software, has blog and wiki modules that are both backed by JCR. We could do the same thing: create version of Apache Roller and Apache JSPWiki (incubating) that share the same content repository.

Later, Jukka Zitting (who also works for Day Software), suggested the idea of implementing JPA itself with JCR, thus allowing Roller to store its content in a CMS in a totally transparent fashion. This topic is interesting to me, but I don't fully understand the benefits of backing blogs and wikis with JCR. What new use cases would this support? How do the interesting features of JCR, like versioning for example, bubble up through Roller -- especially if Roller is to support both RDBMS and CMS back-ends?

And by the way, Day is not the only company building interesting products around JCR and the Apache Jackrabbit JCR implementation. Hippo CTO ArjÃ© Cahn and others from Hippo were present at ApacheCon, talking about their CMS product and related Apache projects.

The REST sessions and the Struts 2 REST plugin

Thursday afternoon there was series of REST talks, starting with Roy Fielding father of REST, then Sanjiva Weerawarana defender of WS-*, Dan Diephouse and Don Brown. Roy's talk was standing room only and the other REST talks were quite popular as well. Hearing from REST masters as great, but Don's talk was the one most relevant to my work (since I'm a Struts 2 user).

Don talked about the new REST plugin he has developed for Struts 2, which looks like a wonderful thing. It builds on the existing CodeBehind plugin to give you Ruby-on-Rails like convention-over-configuration and in most cases no Struts configuration or properties files are necessary. From my point-of-view, it's just another reason to choose Struts 2 over JSF, which, as Bill de hÃ“ra points out, "is clearly not focused on or suitable for working in the REST style to the extent REST principles seem to be actively excluded from the design." See also: Matt Raible's notes on the Struts 2 REST plugin and Don's slides (a zipped PDF file).

Lucene and search sessions

On Friday there was a series of talks related to Apache Lucene and related projects such as Solr. These talks seemed to be quite popular as well. I attended the Solr talk. Apache Solr is a version of the Lucene search engine that is packaged as an easy to install and easy to use web application. The talk demonstrated all of the things you can do with Solr without writing any Java code and it was pretty impressive. I also spoke with Ken Krugler of Krugle, the source code search engine, about search in Roller, something that he's been thing and writing about lately.

Couple of end notes re: Atlanta

First, the venue. The Westin is an impressive building, it's the tallest hotel in the western hemisphere and the rotating bar and restaurant at the top are a lot of fun, with relatively good food and well worth a visit. The hotel itself, though, does not seem to be all that well managed or maintained. The carpets are a little spotty and the furniture is a little worn. Generally, that kind of stuff doesn't bother me much, but I can see why folks might complain (and they did).

Second item: a good dining experience. On Monday night, a big group of hackathon attendees ventured out to look for Rare, billed as a soul-food tapas restaurant. It was quite a long walk from the Westin but we managed to get there with help from Greg Stein's iPhone and some overly helpful folks we ran into on the street. Rare was simply wonderful so next time you're in Atlanta, skip Pittypat's Portch and give Rare a try instead.

And third, MARTA. The travel info I received from the ApacheCon folks recommended taking a shuttle bus from the airport to the Westin, but that's not the best way to go. The MARTA train is a much better way to go -- $4 for a the round trip, it's quick and there's no need to tip.

Wrapping up

That brings me to the end of my notes, so it's time to wrap up. As usual, ApacheCon was a great experience and I hope to make it to the next one in Amsterdam this coming April.

One more thing. You can find many of the session slides on the ApacheCon wiki.

re: JSPWiki@Apache?

2007-09-18T22:26:13+00:00

Congrats are also in order for the JSPWiki team. As Janne Jalkanen notes, JSPWiki was accepted into the Apache Incubator yesterday. I'm proud to be one of the four mentors who will guide the project through the incubation process. Janne and the JSPWiki team put together a great proposal and I think it was clear to everybody involved in the vote that these folks know what they're doing. They'll master the Apache way in no time.

JSPWiki@Apache?

2007-08-07T11:09:38+00:00

Apache JSPWiki Manifesto: This idea has been brought up before, but so far it has not really been an issue. However, this looks like the time when it would be possible to accomplish this.

JSPWiki code base is old, and it needs some refactoring. This refactoring includes things like moving to Java 5, fixing the metadata engine, replacing the backend with something scalable, and in general removing all the cruft that has been accumulated over time. This requires that we break compatibility with existing plugins and other components. Not badly, but to some degree.

Also, JSPWiki as an open source software project is growing slowly but steadily. However, the wiki world is moving rapidly, and wikis have been adopted widely. JSPWiki has become a tool for a great many companies, who are relying on it in their daily business. This is a lot for a hobby project lead by a "benevolent dictator" -model. Therefore, it is time for JSPWiki to mature to a "real" open source software project to be a serious contender in the wiki world.

To accomplish both of these goals needs a major shift in how JSPWiki is managed and who "owns" it, in a sense. Therefore, we (the people who have been committing source code) think that Apache would be a good choice, and have decided that we will try to submit JSPWiki into the Apache incubation process, with the goal of graduating as a top-level project.

I've been a JSPWiki (and Janne Jalkanen) fan for years now. It's my favorite Java-based wiki so I'm pretty excited that the dev team is <a href= "http://ecyrd.com/pipermail/jspwiki-dev/2007-August/000508.html">preparing a proposal to move the project to Apache. I think this will be a great move and will ensure that the project continues to grow and continues to be a strong contender in the enterprise wiki space. I'm more than willing to help with that proposal and to help out in the Incubator.

Apache OpenJPA graduates

2007-05-22T08:50:28+00:00

Congrats to the OpenJPA team.

Patrick Linksey: Last week, OpenJPA graduated from the Apache incubator, meaning that it's now a fully-fledged Apache project. We're graduating to a top-level project, so once the infrastructure administrivia gets worked out, the new URL will be http://openjpa.apache.org.
I'd like to thank Craig Russell for pushing the project through the last mile of graduation details, and our incubation mentors (Brian McAllister, Eddie O'Neil, and Geir Magnusson) for all their help transitioning into the Apache world.

I'm pushing to replace Hibernate with JPA in Roller and OpenJPA seems the most likely candidate, so I'm glad to hear the community is thriving.

ApacheCon EU 2007 wrap-up

2007-05-06T23:09:36+00:00

As usual ApacheCon was a blast. I showed-up on Tuesday, made myself at home in the hackathon room and started reconnecting with old friends and meeting new ones. I arrived at the members reception a little late and missed the beer, but was not too late to meet Lars Trieloff of Mindquarry, a startup that's working on an interesting open source product suite that combines content management, task management and wiki functionality. Behind the scenes the product combines Subversion, Apache Jackrabbit and other open source products. Apparently everything is tied together via the Java Content Repository (JCR) API and that's why Lars is interested in the possibility of hooking Roller up with a JCR backend.

Talks began Wednesday and I sat in the business track for most of the day. I particularly enjoyed Rebecca Hansen's talk Better than free: Strategic opportunities in open source and Bill Stoddards talk on Best Practices for Incorporating Open Source Code in Commercial Production. I also enjoyed Alexandru Popescu talk on Up to Speed with Java Content Repository API and Jackrabbit. I attended Stefano Machacci's excellent Community Building Practices talk again -- I think it should be required for all Apache contributors.

Thursday night was the Sun party at Lloyd Hotel, which was was quite successful. I had an interesting Roller-related chat with Paolo Castagna of HP, who is investigating new ways to integrate blogs, planets and wikis -- so we had a lot to talk about. I'm hoping he'll find that Roller is a good foundation for his work and encouraged him to collaborate with us via the Roller mailing lists. By the way, like Lars Trieloff, he is also interested in JCR as a back-end for blog/wiki data.

My talk Roller and blogs as a web development platform was scheduled for 10:30 Friday morning. It didn't go so well. Power went out at around 10AM and didn't come back until about 20 minutes into the talk. That left me a bit frazzled and feeling rushed, so I don't think I gave my best performance. If you'd like more information on the talk, you can find the outline here and the slides here (1.6MB PDF).

Arrived at ApacheCon EU

2007-05-01T07:03:00+00:00

Vacation is over and it's back to work today. I rode the tram up to Amsterdam Central Station to see Andi off, checked out of our vacation hotel and made my way up to the Movenpick Hotel for ApacheCon EU.

I'm in the ApacheCon hackathon room now working to clear my in-box, catch-up on blogs and figure out what work things need my (rather limited) attention this week.

Vacation was wonderful, by the way. I didn't do any travel blogging on this trip, but I've taken about 200 photos and uploaded some of the best to my Flickr account. I took a bunch of nice shots of the Queens Day festivities yesterday. and the ad hoc boat parades. I've also been building a Google map of our travels, annotated with photos and comments. I'll share that link later this week once it's more complete.