Blogging Roller

Usergrid and Ember.js - part 2

2015-01-26T10:23:01+00:00

In part one, I explained the basics of the example Usergrid-Ember "Checkin" app, how the index page is displayed and how login is implemented. In part two, I'll explain how Ember.js can be hooked into the Usergrid REST API to store and query JSON objects.

Ember.js includes a feature referred to as Ember-Data, which provides a persistence interface for storing and retrieving JavaScript objects that could be stored in memory, or stored on a server and accessed via REST API.

To use Ember-Data with your REST API you've got to define an Ember-Data model and add an Ember-Data REST adapter. If your REST API differs from what Ember-Data expects then you will probably have to extend the built-in REST adapter to handle your URL pattens, and extend the built-in REST serializer to handle your JSON format. By extending Ember-Data in this way, you can use it to store and query data from Usergrid without using the Usergrid JavaScript SDK at all. Below I'll explain what I had to do to make the Checkin app's Activities collection available via Ember-Data.

Define Ember-Data models

Ember-Data expects each of your REST API collections to have a defined data model, one that extends the DS.Model class. Here's what I added for the Activities collection:

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L18" >link)

App.Activity = DS.Model.extend({
  uuid: DS.attr('string'),
  type: DS.attr('string'),
  content: DS.attr('string'),
  location: DS.attr('string'),
  created: DS.attr('date'),
  modified: DS.attr('date'),
  actor: DS.attr('string'),
  verb: DS.attr('string'),
  published: DS.attr('date'),
  metadata: DS.attr('string')
});

Create a custom RESTAdapter

The Ember-Data REST adapter expects a REST API to follow some common conventions for URL patterns and for JSON data formats. For example, if your REST API provides a collection of cats then Ember-Data will expect your REST API to work like so:

What Ember-Data expects for a cats collection:

GET /cats - get collection of cats
POST /cats - create new cat.
GET /cats/{cat-id} - get cat specified by ID.
PUT /cats/{cat-id} - update cat specified by ID.
DELETE /cats/{cat-id} - delete cat specified by ID.

Usergrid follows the above conventions for collections, but there are some exceptions. For example, the Usergrid Activities collection. A GET on the /activities path will return the Activities of the users that you (i.e. the currently authenticated user) follow. You don't POST new activities there, instead you post to your own Activities collection at the path /users/{your-user-id}/activities. It works like this:

Usergrid's Activities collection:

GET /activities - get Activities of all users that you follow.
POST /user/{user-id}/activities - create new Activity for user specified by ID
GET /user/{user-id}/activities - get Activities for one specific user.

To adapt the Activities collection to Ember-Data, I decided to create a new model called NewActivity. A NewActivity represents the data needed to create a new Activity, here's the model:

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L132" >Link)

// Must have a special model for new activity because new 
// Activities must be posted to the path /{org}/{app}/users/activities, 
// instead of the path /{org}/{app}/activities as Ember-Data expects.
App.NewActivity = DS.Model.extend({
  content: DS.attr('string'),
  location: DS.attr('string'),
  actor: DS.attr('string'),
  verb: DS.attr('string')
});

Then, in Checkin's custom REST adapter, I added logic to the pathForType() function to ensure that NewActivities are posted to the correct path. Here's the adapter:

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L40" >Link)

App.ApplicationAdapter = DS.RESTAdapter.extend({

  host: Usergrid.getAppUrl(),

  headers: function() { 
    if ( localStorage.getItem("access_token") ) {
      return { "Authorization": "Bearer " 
          + localStorage.getItem("access_token") }; 
    } 
    return {};
  }.property().volatile(), // ensure value not cached

  pathForType: function(type) {
    var ret = Ember.String.camelize(type);
    ret = Ember.String.pluralize(ret);

    if ( ret == "newActivities" ) {
      // Must have a special logic here for new activity 
      // because new Activities must be posted to the 
      // path /{org}/{app}/users/activities, instead of the 
      // path /{org}/{app}/activities as Ember-Data expects.
      ret = "/users/" + Usergrid.user.username + "/activities";
    }
    return ret;
  }

});

You can see a couple of other interesting things in the example above. First, there's the host field which specifies the base-URL of the REST API for the Checkin app. Next, there's the headers() function, which ensures that every request carries the access_token that was acquired during login.

Create a custom RESTSerializer

Ember-Data also has expectations about the JSON format returned by a REST API. Unfortunately, what Ember-Data expects and what Usergrid provides are quite different. The two examples below illustrate the differences:

Ember-Data vs. Usergrid JSON formats

Ember-Data expects collections like this:

{
   cats: [{
       "id": "6b2360d0",
       "name": "enzo",
       "color": "orange"
   },{
       "id": "a01dfaa0",
       "name": "bertha",
       "color": "tabby"
   }]
}

Usergrid returns collections like this:

{
   action: "get",
   path: "/cats",
   count: 2,
   entities: [{
       "uuid": "6b2360d0",
       "type": "cat",
       "name": "enzo",
       "color": "orange"
   },{
       "uuid": "a01dfaa1",
       "type": "cat",
       "name": "bertha",
       "color": "tabby"
   }]
}

Ember-Data expects individual objects like this:

 
{
   cat: {
       "id": "a01dfaa0",
       "name": "bertha",
       "color": "tabby"
   }
}

Usergrid returns individual objects like this:

 
{
   "id": "a01dfaa0",
   "type": "cat",
   "name": "bertha",
   "color": "tabby"
}

You can see two differences above. Ember-Data expects JSON objects to be returned with a "type key" which you can see above: the "cats" field in the collection and the "cat" field in the individual object. Also, Ember-Data expects an object's ID field to be named "id" but Usergrid returns it as "uuid."

The deal with these differences, the Checkin app extends Ember-Data's DS.RESTSerializer. Here's the code:

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L75" >Link)

App.ApplicationSerializer = DS.RESTSerializer.extend({

  // Extract Ember-Data array from Usergrid response
  extractArray: function(store, type, payload) {

    // Difference: Usergrid does not return wrapper object with 
    // type-key. So here we grab the Usergrid Entities and stick 
    // them under a type-key
    var typeKey = payload.path.substring(1);
    payload[ typeKey ] = payload.entities;

    // Difference: Usergrid returns ID in 'uuid' field, Ember-Data 
    // expects 'id'. So here we add an 'id' field for each Entity, 
    // with its 'uuid' value.
    for ( var i in payload.entities ) {
      if ( payload.entities[i] && payload.entities[i].uuid ) {
        payload.entities[i].id = payload.entities[i].uuid;
      }
    }
    return this._super(store, type, payload);
  },

  // Serialize Ember-Data object to Usergrid compatible JSON format
  serializeIntoHash: function( hash, type, record, options ) {

    // Usergrid does not expect a type-key
    record.eachAttribute(function( name, meta ) {
      hash[name] = record.get(name);
    });

    return hash;
  }
});

In the code above you can see how the extractArray() method moves the "entities" collection returned by Usergrid into a type-key field as expected by Ember-Data and how it copies the "uuid" field to add the "id" field that Ember-Data expects.

We also need to transform the data that Ember-Data sends to Usergrid. You can see this above in the serializeInHash() function, which ensures that when data is POSTed or PUT to Usergrid, the type key is removed because that's what Usergrid expects.

Implementing Add-Checkin

To implement Add-Checkin, I added an HTML template called "add-checkin" to Checkin's index.html file. The template displays an Add-Checkin form with two fields: one for content and one for the location. Here's what it looks like in all its modal glory:

Both fields are simple strings (someday I'd like to extend Checkin to use location information from the browser). I won't go into detail here, but it took a bit of research to figure out how to make a Bootstrap modal dialog work with Ember.js. Below you can see the add-checkin controller, which provides a save() function to save a new checkin.

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L390" >Link)

App.AddCheckinModalController = Ember.ObjectController.extend({

  actions: {

    save: function( inputs ) {

      var content = inputs.content;
      var location = inputs.location;
      var target = this.get("target");

      var activity = this.store.createRecord( "NewActivity", {
        content: content,
        location: location,
        verb: "checkin",
        actor: {
          username: Usergrid.user.username
        }
      });

      activity.save().then(
        function( success ) { 
          alert("Saved"); 
        },
        function( error ) { 
          alert("Error " + error.responseJSON.error_description); 
        }
      ); 

    } 
  }
});

In the code above you can see how easy it is to access Usergrid data via Ember-Data now that we've got our custom REST adapter and serializer in place. We create a new Activity with a call to this.store.createRecord() and to save it all we need to do is activity.save().

Time to wrap up...

To sum things up, here are some closing thoughts and observations.

If you are considering JavaScript MVC frameworks, then Ember.js is definitely worthy of your consideration. The documentation makes it easy to learn and the community is friendly and helpful.
It would be great for Usergrid to provide an Ember.js SDK that makes it really easy to build apps with Ember.js and Usergrid.
Ember-Data is an integral part of Ember.js, something that you need to do pretty much anything, but it is treated as a separate package with separate documentation. That is somewhat confusing for a new user.
Ember-Data does not include built-in form validation so if your app includes a large number of non-trivial forms, then you may prefer AngularJS over Ember.js.
There is a form validation plugin for Ember.js, but it requires the experimental Ember-CLI utility. I tried to use it, but Ember-CLI was unpleasnt enough that I gave up.

I appreciate any feedback you might have about this article, the Usergrid-Ember project and Apache Usergrid. If you want to see how the whole Usergrid-Ember project fits together, find it on GitHub here: Usergrid-Ember. Next up, I'll write about my experiences using Apache Shiro to replace Spring Security in Apache Roller.

Usergrid and Ember.js - part 1

2015-01-20T09:39:47+00:00

The next one of my 2014 Side projects that I’d like to share is Usergrid-Ember, an experiment and attempt to learn more about Ember.js and Apache Usergrid by implementing the Checkin example from my Usergrid mobile development talk. If you're interested in either Usergrid or JavaScript web development then I hope you'll read on...

Why Ember.js?

Ember.js is one of the leading frameworks for building browser-based apps. It's one of many JavaScript Model View Controller (MVC) frameworks. Generally speaking, these frameworks let you define a set of routes or paths in your app, for example /index, /orders, /about, etc. and map each to some JavaScript code and HTML templates. Handling a route usually means using Ajax to grab some “model” data from a server and using a template to create an HTML “view” of the data that calls functions provided in a "controller" object.

JavaScript MVC frameworks are not simple and each has its own learning curve. Is it really worth the learning time when you can do so much with a little library like jQuery? For most projects I think the answer is yes. These frameworks force you to organize your code in a logical and consistent way, which is really important as projects grow larger, and they provide features that may save you a lot of development time.

Based on what I've seen on the net and local meet-ups, the leading frameworks these days are Ember.js and AngularJS. After I saw Yehudi Katz’s talk at All Things Open, I decided to spend some time learning Ember.js.

Getting started with Ember.js

The first thing you see when you visit the Ember.js site is a big button that says "DOWNLOAD THE STARTER KIT" and so that is where I started. The Starter Kit is a, a minimal Ember.js project with about twenty JavaScript, HTML and CSS files. It's a good way to start: small and simple.

Ember.js Starter Kit files:

Sidebar: I do hope they keep the Starter Kit around when the new Ember-CLI tool matures. Ember-CLI generates too many magic boiler-plate files and sub-directories for somebody who is trying to understand the basics of the framework. And this is an interesting point of view: Ember-CLI is Making You Stupid by Yoni Yechezkel.

Other stuff: Bower, Grunt and Bootstrap

I like to bite off more than I can chew, so I decided to use a couple of other tools. I used Bower to manage dependencies and Grunt to concatenate and minify those dependencies, and other things like launching a simple web server for development purposes. I also decided to use Bootstrap to provide various UI components needed, like a navbar and nicely styled list views.

I won't cover the details, but it was relatively easy to get Bower and Grunt working. Here are the config files in case you are interested: bower.json and Gruntfile.js. I did hit one problem: when I included Bootstrap as one of my dependencies the Glyphicons would all appear as tiny boxes, so I decided to pull Bootstrap from a CDN instead (looks like there is a fix for that now).

Defining Index Route, Model and Template

Every Ember.js app needs to define some routes. There is a default route for the "/" path which is called the index route, and you can add your own routes using the Router object. The snippet below shows what I needed to get started:

Part of app.js (link)

// create the ember app object
App = Ember.Application.create();

// define routes
App.Router.map(function() {
    this.route("login", { path: "/login" });  
    this.route("logout", { path: "/logout" });
    this.route("register", { path: "/register" });
});

Ember.js will look for the JavaScript Route and Controller objects as well as the HTML template using the names above. For example: Ember.js will expect the login route to be named App.LoginRoute, the controller to be named App.LoginController and the template to be named "login."

Let's talk about the index route. When a user arrives at your app they’ll be directed to the index route. Ember.js will then look for a JavaScript object called App.IndexRoute to provide the model data and JavaScript functions needed for the index page. Here’s a partial view of the index route:

Part of app.js (link)

App.IndexRoute = Ember.Route.extend( {

    // provide model data needed for index template
    model: function() {
        if ( this.loggedIn() ) {
            return this.store.find("activity");
        }
        return [];
    }

});

The index page of the Checkin app shows the Checkin activities of the people that you follow. Above you can see how to route's model() function makes that data available to the template for display. If the user is logged in we call the store.find(“activity”) function to call the Usergrid REST API to get an array of the latest Activity objects. There is some serious Ember-Data magic going on there and I'll cover that in part two of this article.

To display the index route, Ember looks for an HTML template called “index” and will use that template to display the index page. Below is the index template. The template is a Handlebars template and the things that appear in double curly-braces are Handlebars expressions.

Part of index.html (link)



  
    

      
        Checkin
      

      
          {{action 'showModal' 'add-checkin-modal' model }}>Add Checkin
      

      
        Logout 
      

    
  

  
    

      
        {{#each item in model}}
          
              {{item.content}} | {{item.location}}
          
        {{/each}}

In the above template you can see a couple of {{action}} expressions that call out to JavaScript methods defined in the Checkin app. The part of the code that uses the model is in the {{#each}} loop which loops through each Activity in the model and dispays an HTML list with the the item.content and item.location of each Activity.

Here's what the above template looks like when displayed in a browser:

Implementing Login

In Checkin, login is implemented using HTML Local Storage. Once a user has successfully logged in, the app stores the username and the user's access_token in Local Storage. When user arrives at the index page, we check Local Storage to see if that user is logged in and if not, we direct them to the login route, which in turn displays the login page using the template below.

Part of index.html (link)



  
    
      
        Checkin
      
    
  

  
    
      

        

         Please Login

         Email address
         {{input class="form-control" type="text" 
              valueBinding="username" placeholder="Username"}}

         Password
         {{input class="form-control" type="password" 
             valueBinding="password" placeholder="Password"}}
           
         
            Login
         

         Register as new user.

The LoginController provides the functions needed by the Login page itself and there are two. There is a login() function (called on line 27 above) that performs the login, and there is a register() function (called on line 31 above) that directs the user to the New User Registration page. Here's a snippet of code from the App.LoginController that provides these two functions:

Part of app.js (link)

App.LoginController = Ember.Controller.extend({

  actions: {

    login: function() { 

      // login by POST to Usergrid app's /token end-point

      var loginData = {
        grant_type: "password",
        username: this.get("username"),
        password: this.get("password")
      };

      $.ajax({
        type: "POST",
        url: Usergrid.getAppUrl() + "/token",
        data: loginData,
        context: this,
        error: function( data ) {

          // login failed, show error message
          alert( data.responseJSON.error_description );
        },
        success: function( data ) { 

          // store access_token in local storage
          Usergrid.user = data.user;
          localStorage.setItem("username", loginData.username );
          localStorage.setItem("access_token", data.access_token );

          // clear the form
          this.set("username", ""); 
          this.set("password", "");

         // call route to handle post-login transition
          this.get("target").send("onLogin"); 
        }
      });
    },

    register: function() {
      this.transitionToRoute("register");
    }

  }
});

The above code shows how to login to a Usergrid app using jQuery's Ajax feature. The login() function takes the username and password values from the login form, puts those in a JSON object with grant_type "password" and posts that object to the /token end-point of the Usergrid app. If that post succeeds, the response will include an access_token. We store that in Local Storage; we'll need to use it in all subsequent calls to Usergrid.

Usergrid fans will notice that I'm not using the Usergrid JavaScript SDK. That's because Ember.js provides Ember-Data, which acts as a very nice REST client and can be adapted to work with the URL structure and JSON formats of just about any REST API. I'll write about that in part two of this article.

2014 side projects

2015-01-05T08:37:04+00:00

For various reasons, I've always got a couple of coding projects on the back burner, things that I hack around with on weekends and breaks. In 2014, I started four projects and learned about Ember.js, jQuery Mobile, Apache Shiro, Apache CXF and the Arquillian test framework.

I like to share my code, so I've put my code on GitHub and I'm going to write a brief post about each here on my blog. I'll provide links as I go and, of course, I welcome any criticisms and suggestions for improvement that you might have. First up: the Usergrid-Mobile project.

The Usergrid-Mobile project

To be honest, Budapest was the goal of this project. In the Spring of 2014, I decided that the best chance of getting to ApacheCon EU in Budapest was to create a great "mobile development with Usergrid" talk, and to do that I needed a great example project. The resulting project shows how to create a dumbed-down Foursquare-style "checkin" app using HTML5, JavaScript, jQuery Mobile and Apache Cordova.

Luckily for me, my talk was accepted for ApacheCon EU and in November I traveled to Budapest (took some photos) and gave the talk there.

I also presented the talk at the All Things Open conference in Raleigh, NC and you can view a video of that talk, Mobile Development with Usergrid on YouTube.

You can find the code for usergrid-mobile on GitHub. I also created a Vagrant File to launch a local instance of Usergrid for demo purposes. It's called usergrid-vagrant.

That's all for now. Next up: Usergrid-Ember.

Holiday project: JMaki for Roller

2008-01-02T13:12:54+00:00

Over the holidays I avoided doing anything directly related to my current set of work tasks. Sun went quiet, which helped, and I ignored the messages that piled-up in the Roller user and dev lists. It was so quiet that I had time for a fun little project: a JMaki plugin for Roller.

JMaki makes it easy to use JavaScript widgets (Dojo, Google, YUI, etc.) from PHP, JSP, JSF and now Roller. To use a widget, all you have to do is call a method or include a tag and JMaki takes care of including the right JavaScript files and generating the right HTML for you. That's not all JMaki does, there's also a pub/sub facility to make it easy to wire widgets together via events, there's a proxy for fetching remote resources common table and tree data models. The theme is cool widgets with ease-of-development and that's what I'd like to see in Roller. You can read more about the JMaki value proposition on the Why Use JMaki page.

Here's an example. Below is a Roller page template that uses two JavaScript widgest, the Dojo Clock and the YUI Data Table. All it takes is a single line of template code to include each widget, and one widget is dynamic i.e. the table is populated via an RSS feed.


    <html>
    <head><title>JMaki test page</title></head>
    <body>

        <h1>JMaki test: dojo.clock</h1>
        $jmaki.addWidget("dojo.clock")

        <h1>JMaki test: yahoo.dataTable</h1>
        $jmaki.addWidget("yahoo.dataTable", "/roller/xhp?id=rss","","")

    </body>
    </html>

And here's what that page looks like when displayed by Roller:

I'll write more about the plugin once I install it on this site. If you want some details about how the plugin was developed, you can read the email that I sent to the JMaki dev list: JMaki for Roller issues and suggestions. It links to the Java source code for the plugin.

Pluggable renderers and scripting languages in Roller

2007-03-11T23:45:03+00:00

My next ApacheCon talk is about Roller and blogs as a web development platform. One of the things I plan to discuss is using scripting languages within Roller, something that's possible now because Roller versions 3.0 and later allows pluggable renderers. It's undocumented and a little hacky right now, but by plugging in your own custom renderers you can add support for new template and scripting languages as alternatives to Roller's built-in Velocity. Here are some notes on the whys and hows.

Why would you want to plugin a new scripting or template language?

I can think of two reasons you might want to plug-in a new renderer to Roller. If you're not happy with Velocity and want to plugin an alternative template language like Freemarker you can do it with a custom renderer. Or, if you're developing sophisticated Roller templates and you need a more sophisticated programming language like Groovy, JRuby or Javascript then custom renderers are your best bet.

What can you do in an alternative scripting or template language?

In an alternative template or scripting language you can access the normal Roller model objects (see the Roller Template Guide for a reference) and you can generate just about any type of content you want. However, you cannot access the Roller macros because they're written in and require Velocity.

It's also worth noting that you can get in quite a bit of trouble with an alternative language. Roller's built-in Velocity renderer allows users to access only a very limited set of objects, but other scripting and template languages might not be so safe. That's why you might not want to make all plugin renderers available to all of your users.

How do you implement scripting in a plugin renderer?

I plan on documenting the process at some point, but basically you implement two simple interfaces: RendererFactory and Renderer. So far, I've developed three plugin renderers: BSFRenderer, GroovletRenderer and GSPRenderer. Let's discuss each so you can see how they work.

A BSFRenderer example

The Bean Scripting Framework (BSF) makes it easy to call a wide variety of scripting languages from Java and because it works with Java SE 5, it's a better choice for Roller than the Java SE 6 scripting extensions. I developed a BSF renderer (BSFRendererFactory and BSFRenderer) and with it, it's theoretically possible to support the use of just about any BSF scripting language in a Roller template. I've tried Groovy and JavaScript and they both work fine, but JRuby has some conflicts with the version of Hibernate that we're using in Roller so I'll have to revisit JRuby later.

Unfortunately, with the BSFRenderer you're stuck writing Servlet-like code like this Groovy example:

    out.println("<html><body>")
    out.println("Hello World, my blog is ${model.weblog.name}");
    out.println("</body></html>");

Ugh. Nothing to write home about, that's for sure. Fortunately, there's something better.

A GroovletRenderer example

I also developed a GroovletRenderer and a Groovy Server Pages (GSP) GSPRenderer, which look more promising. These renderers call Groovy directly and behave like the Groovlet and Groovy Template Servlets that come with Groovy.

With my new GroovletRenderer (GroovletRendererFactory and GroovletRenderer), you can use the GroovyMarkup builder to build HTML. Here's an example that displays a blog title, description and most recent entries:

    html.html { // html is implicitly bound to new MarkupBuilder(out)
        head {
            title(model.weblog.name)
        }
        body {
            h1(model.weblog.name)
            i(model.weblog.description)
            map = model.getWeblogEntriesPager().getEntries();
            map.keySet().each() {
               map.get(it).each() {
                  h2(it.title)
                  p(it.text)
                  br()
               }
            }
        }
    }

Starting to look more interesting, no?

A GSPRenderer example

I also developed a GSPRenderer (GSPRendererFactory and GSPRenderer), which allows you to use a JSP like syntax with <% %> and <%= %>. Here's the above example, but rewritten as a Roller GSP page:

    <html>
       <head>
          <title>${model.weblog.name}</title>
       </head>
       <body>
   <h1>${model.weblog.name}</h1>
       <i>${model.weblog.description}</i>
       <% map = model.getWeblogEntriesPager().getEntries();
          map.keySet().each() { %>
              <% map.get(it).each() { %>
                <h2>${it.title}</h2>
                <p>${it.text}</p><br />
              <% }
          }%>
       </body>
    </html>

That's all I have time for tonight. Next I hope tackle JRuby and hopefully I'll find some way to support some form of JRuby templating too. Where is all this going? I don't know. I'm not sure these plugins will make it into the Roller 4.0 release, but I do hope to make it easier to drop them and renderer plugins like them into Roller 4.0.

Roller, JSON and Dojo via JMaki

2007-01-15T23:21:10+00:00

Alexis Moussine-Pouchkine has taken my Generating JSON for your Roller blog post a few steps further down the road. He created a screen-cast to show how easy it is to use Netbeans 5.5 and the JMaki widget wrappers to serve up Roller data in a a Dojo table. Cool stuff. I really need to find the time to take JMaki for a spin.

And BTW, I'm still working on my Dojo table example, which will be similar to what Alexis has done but it will use Dojo directly within a Roller blog -- no IDEs or JSPs required.