Blogging Roller

BlogQL: Storybook and Mocked Service Workers

Dave Johnson — Sat, 8 Apr 2023 20:43:51 +0000

One of the most interesting new technologies I learned about during the development of BlogQL is Storybook. Storybook allows you to develop and test React, Angular, and other web components in isolation.

Once you create stories for your components, and thanks to Chromatic, you can setup GitHub Actions to check each incoming Pull Request for changes to your UI, a visual diff! You can require that a developer approve each change. I found that all pretty compelling and started to develop stories for all of my React components.

The hardest part of this work was figuring out how to mock the GraphQL data needed for my components. It was difficult for me because I was still learning WebPack and polyfills in Node.js, but I got it working, in part thanks to his very useful blog post: Mock GraphQL and REST in Storybook and Jest with MSW.

You can view the BlogQL Storybook online, thanks to Chromatic: https://63b0bf0f112b2d8b80a785f5-flfdjbdgsu.chromatic.com.

This is part of a series of short posts about BlogQL: https://github.com/snoopdave/blogql.

BlogQL, Apollo Server and Google Login

Dave Johnson — Fri, 31 Mar 2023 14:30:00 +0000

The BlogQL server uses Apollo Server to provide a GraphQL API. Initially, I was using Apollo Server standalone, but when I stared integrating Google Login I realized I needed a couple of REST APIs so I added Express https://expressjs.com for that. On the frontend, I'm using React Oauth2 which makes adding Google Login support very easy to implement. This all works fine, but if I were to start over I might consider using a server-side framework like Next.js https://nextjs.org instead of rolling my own Apollo Server and Express setup.

This is part of a series of short posts about BlogQL: https://github.com/snoopdave/blogql.

BlogQL and create-react-app

Dave Johnson — Wed, 29 Mar 2023 17:35:21 +0000

When I first started with BlogQL, I used create-react-app (CRA) to generate the code. It worked well and allowed me to focus on learning React, but eventually I found it too constraining. I wanted to learn the underlying technologies of WebPack and Babel and CRA tucks those away. It took some time, but was able to remove CRA and replace it with my own WebPack and Babel setup.

If I had to start again, I might consider using Vite, which is growing in popularity as an alternative to WebPack.

This is part of a series of short posts about BlogQL: https://github.com/snoopdave/blogql.

BlogQL

Dave Johnson — Mon, 27 Mar 2023 16:20:32 +0000

Last summer, I started a side project to enhance my knowledge of Node, TypeScript, React, GitHub Actions and to dog-food Apollo GraphQL Studio's features that I was working on. In the past couple of months, I've continued to work on BlogQL and now that the UI is not a total embarrassment, I'm going to share a little about the project.

BlogQL is a blog app made up of a TypeScript/Node backend that provides a GraphQL API and a TypeScript/React-based frontend with a rich-text editor for editing blog entries. Below is a two-minute demo so you can see the features that BlogQL supports. You can find the complete source for BlogQL on GitHub https://github.com/snoopdave/blogql.

I'll be following this up with a series of short posts about the technologies I'm using in BlogQL.

Thoughts on ActivityPub, Mastodon, etc.

Dave Johnson — Fri, 25 Nov 2022 01:45:50 +0000

(disclaimer: I'm new at this stuff and do not have full understanding of the AcitvityPub spec so please correct me if you think I got things wrong)

Is the demise of Twitter going to spark a resurgence of the "open web" and is that a good thing?

Context: There is a World Wide Web Consortium (W3C) recommendation (a "standard" in the making) for a social networking service and it is called ActivityPub. If a website implements ActivityPub then it can interact with other websites that also implement ActivityPub. Some are saying 🤣 that ActivityPub based services could be a replacement for Twitter.

What that means is that your social network feed can include people who are on other social networks. For example, if you have an account on a social network that uses ActivityPub, you can follow people who are on any social network that also uses ActivityPub. In the future, from your Instagram account you might be able to follow people on Twitter, Tumblr, Flickr, TikTok, etc. and have them all appear in your feed at Instagram. That all depends on Twitter, Tumblr, Flickr, TikTok and Instagram adding support for ActivityPub, which may or may not happen.

ActivityPub makes social networking more like email. If you have an email account you can send email to anybody regardless of what email service they use. Gmail users can send email to AOL users can send email to Apple iCloud email addresses, etc.

There are already dozens of services that implement ActivityPub and most of them use the Mastodon software. You can find me on Mastodon here: https://mastodon.social/@snoopdave

This all sounds great, but what motivates people, organizations and companies to pay for running ActivityPub servers and how are those entities going to do content moderation on their servers? Will creators be able to get the reach they need to monetize? Will ActivityPub be another enabler of hate speech and dangerous misinformation or is it the foundation on which people can build systems that fight against those things?

Upgraded to Roller 6.1.1

Dave Johnson — Sat, 30 Apr 2022 20:09:11 +0000

A note about this site: I just upgraded rollerweblogger.org to run the recently released Roller 6.1.1, and Java 17. There was one snag. This site uses the Roller-JSPWiki plugin and old Lucene dependency in that plugin prevented Tomcat from loading Roller. It took me a couple of hours to figure out how to upgrade the plugin to use the latest version of JSPWiki. That fixed it.

Building an Open Source J2EE Weblogger

Dave Johnson — Sun, 17 Apr 2022 16:42:50 +0000

I wrote this article for O'Reilly's OnJava.com over twenty years ago and it was published on April 17, 2002. Roller would not become Apache Roller until about five years later. Publishing this article changed my life and set my career on a new trajectory. I can't find it online anymore so to celebrate this anniversay, i'm going to publish it here on Roller.

As a Java developer, you should be aware of the tremendous wealth of open source development software that is available for your use -- even if you have no desire to release any of your own software as open source. In this article, I will introduce you to some of the most useful open source Java development tools by showing you how I used these tools to develop a complete database-driven Web application called Roller.

Roller fits into the relatively new category of software called webloggers: applications that make it easy for you to maintain a weblog, also known as a blog -- a public diary where you link to recent reading on the Web and comment on items of interest to you.

The Roller Web application allows you to maintain a Web site that consists of a weblog, an organized collection of favorite Web bookmarks, and a collection of favorite news feeds. You can define Web pages to display your weblog, bookmarks, and news feeds. By editing the HTML templates that define these pages, you have almost total control over the layout and appearance of these pages. Most importantly, you can do all of this without leaving the Roller Web application -- no programming is required.

I used over a dozen open source development tools to develop Roller, the most useful of which are listed in Table 1; however, this article focuses on just four tools: the XDoclet code generator, the Castor persistence framework, the Struts Servlet/JSP framework, and the Velocity code-generation engine. In this article I will describe the Roller application, its architecture, and specifically how I used XDoclet, Castor, Struts, and Velocity in its development.

Table 1: Open source tools used in Roller Development

Name	Description	Developer	Type of License*
Castor	Persistence framework	Exolab	Similar to BSD license
HSQL	Small but powerful Java database	Thomas Meuller	Similar to BSD license
Jakarta Ant	XML-driven Java build system	Apache	Apache Public License
Jakarta Commons	Collections, utilities	Apache	Apache Public License
Jakarta Struts	Servlet/JSP framework	Apache	Apache Public License
Jakarta Tomcat	Servlet/JSP Server	Apache	Apache Public License
Jakarta Velocity	Template-driven code generator	Apache	Apache Public License
Netbeans	Integrated Dev. Environment		Sun Public License
Xerces	XML parser	Apache	Apache Public License
XDoclet	Code generator	Dreambean	Similar to MIT License

* For more information on open source licenses see opensource.org

The Roller Application

Roller does not support all of the features of commercial weblogging software (such as Userland's Radio or Pyra Labs' Blogger products), but Roller does support what I consider the essential weblogging features. With Roller you can:

Maintain a weblog, with user-defined categories. You can write new weblog entries and edit entries that have already been posted. You can define a set of weblog categories and can assign weblog entries to different categories. This allows you to maintain several different weblogs, each covering a different topic.
Publish your weblog as an RSS news feed. Roller makes your weblog available as a standard Rich Site Summary (RSS) news feed so that readers can subscribe to and read your weblog without visiting your Roller site.
Maintain a collection of favorite bookmarks, organized by bookmark folders. You can define new bookmark folders and can add, delete, and edit the bookmarks within these folders. You can then display these bookmarks on one or more of your Roller site's pages. This allows you to do blogrolling -- displaying links to your favorite weblogs.
Maintain a collection of favorite RSS news feeds. This allows you to display headlines with links to news stories from your favorite news sources or weblogs.
Define a set of Web pages to display your weblog, bookmarks, and news feeds. Pages are defined using HTML templates with embedded macros for each type of data. For example, there is a $Bookmarks macro that will draw a portion of your bookmark collection on a Web page and a $WeblogCalendar macro that will draw a calendar view of your past weblog entries. These templates allow you almost complete control over the layout and look-and-feel of your Web pages.

There are two types of Roller users: readers and editors. Readers are simply anonymous visitors to the Roller Web site. Editors have user accounts and must log in by providing a user name and password. Editors have the ability to edit their weblog entries, bookmarks, newsfeeds, and page templates.

Figure 1 illustrates the Roller application by showing the Roller Web page navigation tree. The boxes represent Web pages and the arrows represent links between pages. The gray pages are the public pages that any visitor may access, the yellow pages are the login pages, and the red pages are the pages that only editors can access.

Figure 1: Roller Web Pages

Roller Architecture

Internally, Roller is divided into a presentation tier and a business tier, as recommended in Sun's J2EE Pattern Catalog. The presentation tier is responsible for Roller's user interface, and the business tier is responsible for Roller's application logic and the persistence of application data. Figure 2 provides an overview of the Roller architecture.

Figure 2: Roller Architecture

The presentation tier is implemented using the Model-View-Controller (MVC) pattern and the Struts MVC framework. The Model is an abstraction of the application logic and application data and is represented by a set of interfaces defined in the org.roller.model package. The View is implemented using Servlets, JSP pages, and Velocity page templates. The Controller is Struts, which is responsible for receiving incoming requests and dispatching them to the View. The implementation of the presentation tier is further discussed in the sections on Struts and Velocity.

The business tier implements the interfaces in the org.roller.model package, using the Castor JDO persistence framework. The business tier exchanges data with the presentation tier in the form of simple, lightweight JavaBeans known as Value Objects. Value Objects are yet another of the Sun J2EE patterns. Each Value Object maps to a table in the Roller database.

Figure 3 shows the Roller Value Objects, their properties, and the relationships between them. Each editor is represented by a User object. Each User has a Website object, which represents the editor's Web site and which has weblog entries, bookmark folders, newsfeeds, and page templates. The Website object also specifies the default page template of the Web site and which page template is used for rendering a day of weblog entries.

Figure 3: Roller Value Objects

The business tier uses Castor JDO to store and retrieve Value Objects to and from a JDBC-accessible database. Castor JDO is part of the larger Castor data-binding framework, which according to the Castor Web site is "the shortest path between Java objects, XML documents, SQL databases, and LDAP."

As a persistence framework, Castor JDO is similar to commercial object-relational mappers such as TopLink and Cocobase. Castor JDO fulfills a role similar to that of Sun's Java Data Objects, but Castor JDO is not an implementation of Sun's JDO specification (JSR-000012). Castor JDO allows you to define a mapping between Java classes and tables in a relational database. You can then issue queries using Castor's own Object Query Language (OQL) and receive the results as collections of Java objects.

Before you can use Castor JDO, you must provide a mapping file -- an XML file that maps each class to a database table and each class property to a field within a database table. Below is a portion of Roller's mapping file.

<mapping> 
<class name=org.roller.model.BookmarkData" identity="id"
    access="shared" key-generator="UUID" auto-complete="false">
    <map-to table="bookmark"/> 
    <cache-type type="count-limited"/>
    <field name="folderId" type="java.lang.String"></field>
    <field name="id" type="java.lang.String"></field>
    <field name="image" type="java.lang.String"></field>
    <field name="name" type="java.lang.String"></field>
    <field name="priority" type="java.lang.Integer"></field>
    <field name="url" type="java.lang.String"></field>
</class>
...
</mapping>

Once you provide Castor with a mapping file, retrieving a collection of objects from the database can be as simple as the code snippet shown below:

// Construct a new query and bind its parameters
String query = "SELECT p FROM BookmarkData p WHERE websiteId=$";
OQLQuery oql = db.getOQLQuery( query );
oql.bind( websiteId );

// Retrieve results and print each one
QueryResults results = oql.execute();
while ( results.hasMore() ) {
   BookmarkData bookmark = (BookmarkData)results.next();
   System.out.println( bookmark.toString() );
}

XDoclet

XDoclet is a code generator that is implemented as a Javadoc extension, a Doclet. To use XDoclet, you place special Javadoc tags in your Java source code. Based on these tags, XDoclet can generate additional Java code that supports your classes, mapping files that map your classes to database tables, and deployment descriptors that assist in deploying your classes.

XDoclet started out its life as EJBDoclet, a tool that allows you to implement an Enterprise JavaBean by writing just one source code file. Now, the XDoclet product includes two Doclets: EJBDoclet and WebDoclet. EJBDoclet is for generating EJB classes, value objects, and database mappings. WebDoclet is for generating all sorts of Servlet Web Application deployment descriptors, including web.xml files, Tag Library Descriptors, and Struts configuration files.

The Roller build process uses both EJBDoclet and WebDoclet, as shown in Figure 4. In Step 1, EJBDoclet is used to process a set of abstract classes of type javax.ejb.EntityBean -- one for each one of the Roller Value Objects. From these classes, EJBDoclet generates a Castor mapping file, the Roller Value Object classes, and a set of corresponding Struts form classes. In Step 2, WebDoclet is used to process a source directory that contains JSP tags, Servlet classes, and Struts classes. The output of the WebDoclet is the complete set of Roller Web Application deployment descriptors.

Figure 4: XDoclet and the Roller Build Process

Below is a simple example bean that shows the EJBDoclet tags necessary to create a Value Object. The @castor tags provide the information needed to generate the Castor mapping entries for the bean. The @ejb tags provide the information needed to generate the Value Object and a complete EJB entity bean (which Roller does not use).

/**
 * Represents a single URL in a user's favorite web-bookmarks collection.
 * @ejb:bean name="Bookmark" type="CMP" jndi-name="roller/Bookmark"
 * @ejb:data-object extends="org.roller.model.ValueObject"
 * @struts:form 
 * @castor:class name="bookmark" table="bookmark" xml="bookmark"
 *               id="id" key-generator="UUID"
 */

public abstract class BookmarkBean implements EntityBean 
{

  /** @ejb:interface-method 
   *  @ejb:transaction type="Required" */
   public abstract void setData(org.roller.model.BookmarkData dataHolder);

  /** @ejb:interface-method */
   public abstract org.roller.model.BookmarkData getData();

  /** @castor:field set-method="setId"
   *  @castor:field-xml node="attribute"
   *  @castor:field-sql name="id" sql-dirty="check" dirty="true" 
   *  @ejb:interface-method 
   *  @ejb:pk-field
   *  @ejb:persistent-field */
    public abstract String getId();

  /** @ejb:pk-field
   *  @ejb:persistent-field */
    public abstract void setId( String value );
    ...
}

Struts

The Roller presentation tier is implemented using Struts and Velocity. Struts is a Servlet application framework that is based on the MVC pattern. In a typical Struts application, the Model is a set of JavaBeans that hold the data to be presented in the View; the View is a set of JSP pages that render HTML; and the Controller is a Servlet and set of action classes that are registered to handle incoming requests.

Roller's Edit-Bookmark form provides a nice, simple example of how Struts works. There are four parts to the Edit-Bookmark form implementation: the edit-bookmark.jsp page, the BookmarkForm JavaBean class, the BookmarkFormAction action handler, and some entries in Roller's struts-config.xml file that tie the first three items together. So, let's introduce the players:

The edit-bookmark.jsp page looks just like an HTML page, except that it uses the Struts HTML form tags instead of standard HTML form tags. The Struts HTML form tags know how to find the BookmarkForm JavaBean and how to use its properties to populate the form with data.
The BookmarkForm class is a dumb JavaBean that just holds data -- it has the exact same properties as the Bookmark Value Object. As you may recall, the BookmarkForm class and all of its sibling form classes are generated by XDoclet. In Struts, form classes must extend org.apache.struts.action.ActionForm.
The BookmarkFormAction is essentially an action handler. It is registered (in the struts-config.xml file) to handle incoming requests that include the pattern /bookmark.do. In Struts, action classes must extend org.apache.struts.action.Action.

Figure 5 shows the sequence of events that occurs when a request for the Edit-Bookmark form comes into the system. Roller needs to respond to this request by creating an HTML form populated with data for the bookmark that is to be edited.

Figure 5: Incoming request for Edit-Bookmark page

Here are the steps in processing an incoming request for the Edit-Bookmark page:

The Struts Controller Servlet receives a request for the Edit-Bookmark action. The Controller uses the URI of the request to look up the FormAction that should handle the request.
The Struts Controller Servlet dispatches the request to the BookmarkFormAction.edit() method. Knowing that the user has requested the Edit-Bookmark page, the BookmarkFormAction looks for a request parameter that specifies the bookmark that is to be edited.
The BookmarkFormAction calls the BookmarkManager to retrieve the bookmark information that is to be edited.
The BookmarkFormAction creates the BookmarkForm bean and adds that bean to the request's attributes so that it can be accessed by the JSP page.
The BookmarkFormAction finally forwards the request to edit-bookmark.jsp so that the page may be rendered.
The Struts form tags on the edit-bookmark.jsp page reads data from the BookmarkForm bean and uses that data to populate the Edit-Bookmark form. After that, the HTML page is returned to the user's browser for display.

Figure 6 shows the sequence of events that occurs when the request that contains posted data from the Edit-Bookmark page comes into the system. Roller needs to take the incoming form data and use it to update the bookmark that is stored in the data store managed by the business tier.

Figure 6: Request with data posted from Edit-Bookmark page

Here are the steps in processing a request with data from a posted Edit-Bookmark page:

The Struts Controller Servlet receives a request for the Update-Bookmark action. The Struts Controller determines which action should handle the request and which form bean should receive the data from the incoming form post.
The Struts Controller Servlet populates the BookmarkForm bean with data from the incoming request.
The Controller calls the BookmarkFormAction and passes in the form bean.
The BookmarkFormAction retrieves the data from the BookmarkForm bean.
The action calls upon the BookmarkManager to store the updated bookmark information.

Velocity

While JSP pages work well for the Roller editor pages, which rarely change, JSP does not work so well for the user pages. Weblog authors are not programmers, and they cannot be required to learn JSP and Java programming just to customize their weblog and associated Web pages. Furthermore, allowing Roller users to add new JSP pages, and thus new Java code, to the Roller application at runtime is a security risk.

Figure 7: Velocity-generated public page

The best solution to the user pages problem is Velocity. Velocity is a general purpose template-based code-generation engine. That may sound complicated, but from the user's point of view, it is simple and easy-to-use. For example, the weblog page shown in Figure 7 is generated by a simple Velocity template. This template is shown below:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
;<head>
<title>$macros.showWebsiteTitle()</title>     
<style type="text/css">$macros.includePage("_css")</style>
</head>
;<body> 
<table cellpadding="5" cellspacing="15" 
   border="0" align="center" width="95%">   
   <tr>    
   <td width="20%" valign="top" bgcolor="#ffffff">      
      $macros.showNavBar(true)<br>       
      $macros.showEditorNavBar(true)<br>
      $macros.showBookmarks("Blogrolling",true)<br>       
      $macros.showBookmarks("News",true)    
   </td>    
   <td width="60%" valign="top" bgcolor="#ffffff">                
      <h2>$macros.showWebsiteTitle()</h2>                       
      $macros.showWeblogCategoryChooser()<br>       
      $macros.showWeblogEntries()    
   </td>    
   <td valign="top" bgcolor="#ffffff" width="20%">         
      $macros.showWeblogCalendar()<br>       
      $macros.showRSSBadge()    
   </td> 
   </tr>
</table>                  
</body>
</html>

The items that start with $ are Velocity expressions, most of which result in calls to JSP tags that have been specially designed to work with Velocity. For example, the $macros.showWeblogCategoryChooser() expression results in the generation of the navigation bar at the top of the page -- the one that reads "All | Technology | News | Entertainment." The navigation bar is implemented in a custom JSP tag class named org.roller.presentation.tags.NavigationTag, which is also used in the JSP-based Roller editor pages.

Each user can define any number of pages, and since these pages are simply HTML pages, they can be customized using Front Page or any other HTML editor. The user just has to put the Velocity expressions in the right place. Below is a list of some of the Velocity expressions that are available for use in user-defined Roller Web pages.

Macro	Emits HTML for:
$macros.showNavBar()	Navigation bar, with a link to each one of the user's user-defined pages
$macros.showEditorNavBar()	Editor navigation bar, with links to the edit-bookmarks,edit-newsfeeds, edit-weblog, and edit-website pages
$macros.showBookmarks()	Entire bookmark collection in a multi-column table
$macros.showNewsfeeds()	Current headlines and story descriptions for the user's RSS newsfeeds
$macros.showWeblogEntries()	The most recent weblog entries
$macros.showWeblogCalendar()	A weblog calendar, with a link for each day on which there is a weblog entry

Conclusion

In this article, I have described four open source Java development tools and how these tools can be used together to develop a fairly sophisticated Web application. I hope I have given you a good idea of the power and flexibility of these tools.

Although I have not mentioned any problems with the open source tools that I have discussed, I did run into a number of bugs. I was able to find work-arounds and fixes for these bugs, but it was not always easy. I had to spend some time browsing mailing-lists, searching with Google, and, in one case, downloading the latest source for a product and building it myself. Formal technical support is not available for many open source tools, so keep in mind that you may have to solve your own problems.

In closing, I would like to thank the many developers and other contributors that made possible the open source Java development tools that I used in the development of Roller. The tools are great and they just keep getting better.

Resources

Weblogging

Blogging about Roller 6.1

Dave Johnson — Wed, 22 Dec 2021 22:53:38 +0000

I barely even update this blog, but I do update the software that powers it and I'm happy to still have the help of an awesome team of volunteers who pitch in when they want to and always when needed. Today we released Apache Roller 6.1.0, a release we had been talking about, but that was prompted by the Log4J vulnerability. See the project's blog for details.

I'm not the one making the most code changes in Roller now days, but I do help with releases. I've been spending my spare cycles hacking on BlogQL, a Node/TypeScript-based blog server with a GraphQL API and a React front-end. It's really more of an example app to help me understand those technologies, kind of like Roller was. Maybe I'll write about it someday. That's all for now.

Apache Roller 6.0.2 released!

Dave Johnson — Tue, 17 Aug 2021 01:38:55 +0000

The Roller PMC has approved the release of Apache Roller 6.0.2, a minor bug fix release.

You can download the release via Apache mirrors link here:

http://roller.apache.org/downloads/downloads.html

Roller 6.0.2 includes the fixes listed below:

Parse referrer URL instead of using string value directly

https://github.com/apache/roller/pull/90

Improved: the UI for the blog entries page

https://github.com/apache/roller/pull/87

Improved: tooltip content on the registration form

https://github.com/apache/roller/pull/89

Updated: the main menu to have proper space when displaying permission message

https://github.com/apache/roller/pull/95

Updated: the header to show the tagline only when it's available

https://github.com/apache/roller/pull/99

UI fixes are thanks to new contributor Yash Maheshwari.

On behalf of the Roller PMC,

Enjoy and best regards!

Roller 6 released

Dave Johnson — Tue, 31 Dec 2019 15:04:17 +0000

This latest release of Roller includes a new UI that uses Twitter Bootstrap 3 and is based on work I started in 2015 and first committed on December 21, 2015 with this commit: 2da6c3c2e28419f68244e0c362c15be96013d5f9. You can find the details on on the Roller project blog. I got lot of help along the way with testing, fixes, dependency upgrades, Java 11 support and more, so thanks to all that helped make this happen.

Powered by Digital Ocean Kubernetes

Dave Johnson — Sun, 10 Feb 2019 22:42:05 +0000

Just a note to say that I've switched this site over to Digital Ocean Kubernetes service, which is in Limited Availability right now.

Digital Ocean's Kubernetes service is just as simple and well designed as the rest of Digital Ocean. I mentioned before that I rolled my own Kubernetes cluster via Ansible and Kubeadm. Now I can delete all those config files and that's a good thing. Plus, the price is right; I can get by with one $10/month node (1 CPU / 2 GB memory) and a $10/month load balancer.

To get this site up and running I had to deploy four things to my cluster. I installed the NGINX Ingress Controller, Cert-Manager for automatic creation of Let's Encrypt TLS certs, PostgreSQL and my custom build of Apache Roller. All of that went pretty smoothly and I didn't run into and problems that I could blame on Digital Ocean.

Roller 6.0.0-SNAPSHOT

Dave Johnson — Sat, 26 Jan 2019 22:13:38 +0000

Upgraded this site to Roller 6.0.0-SNAPSHOT today, which meant an hour of fiddling around with my private Docker registry, then giving up and using the one free private repository offered by DockerHub and then, another hour of futzing around trying to figure out my PostgreSQL JDBC driver doesn't work anymore (I inadvertently upgraded from JDK 1.7 to 1.8) and why I can't seem to upgrade it (Kubernetes caches Docker images unless you set imagePullPolicy to always). In the end, I got it working. This post is written in the yet to be officially release Apache Roller 6.0.0-SNAPSHOT version.

Side note: the new rich-text editor in Roller is now Summernote and it seems quite nice. I need to tweak it a bit because there is currently no way to set the font or add a link unless you switch to raw HTML mode.

Roller's new web UI

Dave Johnson — Mon, 21 Jan 2019 15:59:00 +0000

About three years ago I decided to modernize and improve the Apache Roller web UI by rewriting the JSP pages to use the Struts 2 Bootstrap tags, which use Twitter's Bootstrap v3 components and JavaScipt. I also wanted to replace all the HTML table-based formatting with div's and Bootstrap, do a bunch of other improvements and make Roller's web UI less clunky and annoying.

Converting Roller's eight-five JSP pages was a big task and I did not have much time for it. That's why it took three years. Ironically, the Roller modernization project leaves Roller three years out of date. Still, I think it is a huge improvement over the Roller v5 web UI and I want to get it released in Roller v6. Currently, this work is available as Pull Request #22 and you can find some screenshots there too. Here's one:

Try it with Docker-Compose

I also did some work to make it super-easy to try the Roller v6 snapshot pre-release for yourself, by using Docker Compose. You don't have to fiddle with Tomcat or PostgreSQL. You can find a simple Dockerfile for running Roller v2 snapshot and a docker-compose.yml file linked below. And you can find a Docker image in my DockerHub repo.

If you want to try Roller v6 snapshot, here's what you need to do:

1 - If you don't aleady have it, install Docker

2 - Create a directory on your computer where you want Roller to store it's data.

3 - Save this file docker-compose.yml to that new directory.

4 - Open a shell in that new directory and run:

docker-compose up

5 - Watch the PostgreSQL and Roller startup logs scroll by

6 - When the log scroll slows go to http://localhost:8080 to access Roller and go through the initial setup.

Alternatively, if you want to try Roller the hard way, you can get the regular-style v6 SNAPSHOT release files here roller/roller-6.0/v6.0.0.

Let us know how it goes

I hope you'll give Roller v6 snapshot a try and let the project know how it can be improved for your use. Send feedback to the Roller mailing lists or ttweet at us at @apache_roller.

Powered by Kubernetes

Dave Johnson — Tue, 13 Mar 2018 14:29:55 +0000

Just a quick note to say that I ditched Docker Swarm and now this rarely updated blog is powered by Kubernetes. Total overkill, I know. Like Roller itself, I did it as a learning exercise. I hope to blog more about what I learned by doing this. For now, here's a quick summary of what I've done so far.

Created a cluster

I created a 2-node Kubernetes cluster on Digital Ocean using some hand-crafted Ansible scripts that call apt-get to install and kubeadm to start Kubernetes. I considered using Typhoon to create the cluster, but I really wanted to learn how to install Kubernetes "from scratch".

Ran two Ingress Controllers

To avoid using Digital Ocean's $20/month load balancer I'm running an Nginx Ingress controller on each node, and pinning containers to nodes using labels and nodeSelectors. I had to borrow Nginx Controller setup files from the Typhoon project because I'm still kind of bewildered by Ingresses.

Deployed my containers

Next, I wrote Kubernetes YAML files for deploying my containers: a private Docker Registry, PostgreSQL and my custom Roller image. Getting the private registry working properly was the biggest challenge. I need private because I don't want to make my custom Roller image public. Next, I'll install Jenkins next for CI/CD of my custom Roller build via the Jenkins Kubernetes plugin.

Let me know if there are any aspects of this that you'd like to see covered in a blog entry, or suggestions for running the cluster without two Ingress Controllers. I've already got a post cooking about installing a TLS secured Docker Registry on Kubernetes.

Powered by Postgresql and Docker Swarm

Dave Johnson — Tue, 7 Nov 2017 21:52:40 +0000

It was somewhat painful but due to some problems with MySQL and Docker, and some general uneasiness with MySQL, I switched this site from MySQL v5.7 to PostgreSQL v10. I also switched over to Docker Swarm. Here's the Docker-Compose file that I'm using now to run this site:

version: '3.2'

services:

   postgresql:
      image: "postgres:10.0"
      ports:
         - "5432:5432"
      deploy:
         resources:
           limits:
              memory: 50M
      volumes:
         - type: bind
           source: /var/lib/postgresql/data
           target: /var/lib/postgresql/data
      environment:
        - POSTGRES_USER=roller
        - POSTGRES_DB=rollerdb
        - POSTGRES_PASSWORD_FILE=/run/secrets/pg_passwd
      secrets:
        - source: db_passwd
          target: pg_passwd

   roller:
      image: "rwo:latest"
      ports:
        -  "80:8080"
      depends_on:
        - postgresql
      deploy:
         resources:
           limits:
              memory: 800M
      volumes:
        - type: bind
          source: /var/lib/roller
          target: /var/lib/roller
      environment:
        - DB_HOST=postgresql
        - STORAGE_ROOT=/var/lib/roller
        - JAVA_OPTS="-Xmx700m"

secrets:
  db_passwd:
    file: ./db_passwd.txt

It was a pain, but sometimes pain = gain and I learned a lot. I'm hoping the site will be a bit more stable now.

Modernizing the Roller UI

Dave Johnson — Sat, 11 Jun 2016 18:28:36 +0000

I don't blog very often but I still find time to work on my blog's software: Apache Roller.

Recently, I decided to focus on improving Roller's ancient Struts 2-based user interface (UI). I had considered adding a comprehensive API to Roller and building a new UI based on that API, but wow that is a huge amount of work. Instead, I decided to modernize the Roller UI by using Twitter's Bootstrap components and CSS styles.

So far, I've devoted a couple of weekends to this work and made some pretty good progress. I'm about half-way done. I'm using the Struts2-Bootstrap plugin, adding better client-side form validation with JavaScript and doing my best to improve the overall user experience. You can see an album of the pages I've done so far on Flickr:
Roller UI with Bootstrap.

I would love any contributions, so if you are interested in helping out, please submit Pull Requests against the bootstrap-ui branch in the Apache Roller repo on GitHub.

Fractal fun with HTML5 and TypeScript

Dave Johnson — Mon, 11 Jan 2016 10:00:00 +0000

MbCanvas is a fun project that I did in 2015: a simple Mandelbrot Set viewer written in Typescript and using the HTML5 Canvas. I did the project to learn more about Typescript and the HTML5 Canvas and I must say, Typescript very nice -- so much easier to read and write than plain old JavaScript, at least for me.

Here's an example image from the viewer.

The project is fairly easy to build if you've got Node and NPM installed, or you can play around with it here: mbcanvas - Mandelbrot viewer in TypeScript

Usergrid-Vagrant updated for Usergrid 2

Dave Johnson — Sat, 2 Jan 2016 11:44:45 +0000

If you're interested in trying the not-yet-released Apache Usergrid 2 you might want to checkout my Usergrid-Vagrant project on GitHub. I just updated the project to support Usergrid 2, using the latest code from the Usergrid "release" brach. The big changes were switching to OpenJDK 8 and adding ElasticSearch. I also rewrote the scripts to use plain old Bash instead of Groovy.

https://github.com/snoopdave/usergrid-vagrant

If you want the old Usergrid 1 Vagrant-file then checkout the "1.x" branch.

Now hosted on DigitalOcean

Dave Johnson — Sat, 2 May 2015 13:46:00 +0000

After thirteen years of hosting this blog at Kattare.com, I've moved it over to DigitalOcean. Kattare was great, but nowadays I prefer managing my own server and DigitalOcean makes that very easy -- and costs less ($10/month vs. $26/month at Kattare).

The move was easy, or as easy as setting up OpenJDK 8, Tomcat 7 and MySQL 5.5 can be. I only hit one little snag. Once I added the Roller WAR to Tomcat, Tomcat would hang on startup. I used jstack to look at the Java VM threads and found some clues that led me to a post on ServerFault.com: Tomcat 7 hangs on deploying apps. As recommended in that post, I added -Djava.security.egd=file:/dev/./urandom to my CATALINA_OPTS and was back in action.

Apache Shiro for authentication in Roller

Dave Johnson — Mon, 9 Feb 2015 07:27:50 +0000

This is the third of my 2014 side projects that I'm sharing and one that involves the Apache Roller blog server and the Apache Shiro security framework. You might find this interesting if you're considering using Shiro for authentication and authorization, or if your interested in how security works in Apache Roller.

Inspired by my work with Ember.js in Fall 2014, I started thinking about what it would take to build an Ember.js-based editor/admin interface for Apache Roller. To do that, I'd need to add a comprehensive REST API to Roller, and I'd need a way to implement secrity for the new API. I've enjoyed working with Apache Shiro, so I decided that a good first step would be to figure out how to use Apache Shiro in Roller for Roller's existing web interface.

Working over the winter break I was able to replace Roller's existing Spring security implementation with Shiro and remove all Spring dependencies from my Rollarcus fork of Roller. Below I'll describe what I had to do get Shiro working for Form-base Authentication in Roller.

Creating a Shiro Authorizing Realm

The first step in hooking Shiro into Roller is to implement a Shiro interface called ShiroAuthorizingRealm. This interface enables Shiro to do username and password checks for users when they attempt to login, and to get the user's roles.

Below is the first part of the class, which includes the doGetAuthenticationInfo() method, which returns the AuthenticationInfo for a user specified by an AuthenticationToken that includes the user's username. In other words, this method allows Shiro to look-up a user by providing a username and get back the user's (hashed) password, so that Shiro can validate a user's username and password.

ShiroAuthorizingRealm.java (link)

public class ShiroAuthorizingRealm extends AuthorizingRealm {

    public ShiroAuthorizingRealm(){
        setName("ShiroAuthorizingRealm");
        setCredentialsMatcher(
            new HashedCredentialsMatcher(Sha1Hash.ALGORITHM_NAME));
    }

    @Override
    public AuthenticationInfo doGetAuthenticationInfo(
        AuthenticationToken authToken) throws AuthenticationException {

        UsernamePasswordToken token = (UsernamePasswordToken) authToken;

        User user;
        try {
            user = loadUserByUsername( token.getUsername() );

        } catch (WebloggerException ex) {
            throw new AuthenticationException(
                "Error looking up user " + token.getUsername(), ex);
        }

        if (user != null) {
            return new SimpleAuthenticationInfo( 
                user.getUserName(), user.getPassword(), getName());

        } else {
            throw new AuthenticationException(
                "Username not found: " + token.getUsername());
        }
    }

In the code above you can see how we pull the username out of the authToken provided by Shiro and we call a method, loadUserByUserName(), which uses Roller's Java API to load a Roller user object specified by name.

The next method of interest is doGetAuthorizationInfo(), which allows Shiro to look-up a user's Role. This allows Shiro to detmerine if the user is a Roller admin user or a blog editor.

ShiroAuthorizingRealm.java (continued)


    public AuthorizationInfo doGetAuthorizationInfo(
        PrincipalCollection principals) {

        String userName = (String)
            (principals.fromRealm(getName()).iterator().next());

        User user;
        try {
            user = loadUserByUsername( userName );
        } catch (WebloggerException ex) {
            throw new RuntimeException("Error looking up user " + userName, ex);
        }

        Weblogger roller = WebloggerFactory.getWeblogger();
        UserManager umgr = roller.getUserManager();

        if (user != null) {
            List roles;
            try {
                roles = umgr.getRoles(user);
            } catch (WebloggerException ex) {
                throw new RuntimeException(
                    "Error looking up roles for user " + userName, ex);
            }
            SimpleAuthorizationInfo info = new SimpleAuthorizationInfo();
            for ( String role : roles ) {
                info.addRole( role );
            }
            log.debug("Returning " + roles.size() 
                + " roles for user " + userName + " roles= " + roles);
            return info;

        } else {
            throw new RuntimeException("Username not found: " + userName);
        }
    }

In the code above you can see that we use the loadUserByUsername() too look-up a user by username, then we use Roller's Java API to get the user's roles. We add those roles to an instance of the Shiro class SimpleAuthorizationInfo and return it to Shir.

Creating a Shiro Authorizing Filter

Now that we've implementated a realm, we've provided Shiro with everything needed to authenticate Roller users and get access to Roller user role information. Next, we need to configure Shiro to enforce roles for the URL apths found in Roller. Shiro includes a RolesAuthorizationFilter, which is close to what we need but not exactly right for Roller. I had to extend Shiro's roles filter so that we can allow a user who has any (not all) of the required roles for a resource.

RollerRolesAuthorizationFilter.java (link)

public class RollerRolesAuthorizationFilter 
    extends RolesAuthorizationFilter {

    @Override
    public boolean isAccessAllowed( 
        ServletRequest request, 
        ServletResponse response, 
        Object mappedValue) throws IOException {

        final Subject subject = getSubject(request, response);
        final String[] roles = (String[]) mappedValue;

        if (roles == null || roles.length == 0) {
            return true;
        }

        // user is authorized if they have ANY of the roles
        for (String role : roles) {
            if (subject.hasRole(role)) {
                return true;
            }
        }
        return false;
    }
}

Configuring Shiro for Roller

Now that we've seen the Java code needed to hook Shiro into Roller, lets look at how we configure Shiro to use that code. We do that using the Shiro configuration file: shiro.ini, as shown below.

shiro.ini (link)

[main]

defaultRealm = org.apache.roller.weblogger.auth.ShiroAuthorizingRealm
securityManager.realms = $defaultRealm

cacheManager = org.apache.shiro.cache.ehcache.EhCacheManager
securityManager.cacheManager = $cacheManager

authc = org.apache.shiro.web.filter.authc.FormAuthenticationFilter
authc.loginUrl = /roller-ui/login.rol
authc.successUrl = /roller-ui/menu.rol

rollerroles = org.apache.roller.weblogger.rest.auth.RollerRolesAuthorizationFilter

[urls]

/roller-ui/login.rol          = authc
/roller-ui/login-redirect.rol = authc, rollerroles[admin,editor]
/roller-ui/profile**          = authc, rollerroles[admin,editor]
/roller-ui/createWeblog**     = authc, rollerroles[admin,editor]
/roller-ui/menu**             = authc, rollerroles[admin,editor]
/roller-ui/authoring/**       = authc, rollerroles[admin,editor]
/roller-ui/admin/**           = authc, rollerroles[admin]
/rewrite-status/**            = authc, rollerroles[admin]
/roller-services/rest/**      = authcBasic, rollerroles[admin,editor]

In the configuration file above, you see how we hook in the new ShiroAuthorizingRealm on line 3. The next couple lines are boiler-plate code to hook in Shiro's caching mechanism and then, on line 9, we configure an authentication method called authc, which is configured to use Shiro's Form Authentication feature. And, on line 13, we hook in our new RollerRolesAuthorizationFilter.

Next, we tell Shiro that the login page for Roller is /roller-ui/login.rol and which page to direct a user to on a successful login, /roller-ui/menu.rol, if the user did not specify which page they wanted to access. And finally, on lines 17-25, you see the list of Roller URL patterns that need protection, which authentication method to use (authc or authcBasic) and the authorization filter and roles required for access to the URL pattern.

Wrapping up...

That's all there is to the story of Roller and Shiro so far. I was able to get Roller's form-based authentication working with Shiro, but I did not try to test with OpenID or LDAP, so I assume more work will be necessary to get them working. I did the work in my experimental Rollarcus fork of Roller. You can get the code from the shiro_not_spring branch. Pull requests are quite welcome as are suggestions for improvement. Please let me know if you see anything wrong in the above code.

This work may not find its way into Roller proper, but it plays a part in my the next side-project that I will share: A REST API for Roller with JAX-RS.

Usergrid and Ember.js - part 2

Dave Johnson — Mon, 26 Jan 2015 10:23:01 +0000

In part one, I explained the basics of the example Usergrid-Ember "Checkin" app, how the index page is displayed and how login is implemented. In part two, I'll explain how Ember.js can be hooked into the Usergrid REST API to store and query JSON objects.

Ember.js includes a feature referred to as Ember-Data, which provides a persistence interface for storing and retrieving JavaScript objects that could be stored in memory, or stored on a server and accessed via REST API.

To use Ember-Data with your REST API you've got to define an Ember-Data model and add an Ember-Data REST adapter. If your REST API differs from what Ember-Data expects then you will probably have to extend the built-in REST adapter to handle your URL pattens, and extend the built-in REST serializer to handle your JSON format. By extending Ember-Data in this way, you can use it to store and query data from Usergrid without using the Usergrid JavaScript SDK at all. Below I'll explain what I had to do to make the Checkin app's Activities collection available via Ember-Data.

Define Ember-Data models

Ember-Data expects each of your REST API collections to have a defined data model, one that extends the DS.Model class. Here's what I added for the Activities collection:

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L18" >link)

App.Activity = DS.Model.extend({
  uuid: DS.attr('string'),
  type: DS.attr('string'),
  content: DS.attr('string'),
  location: DS.attr('string'),
  created: DS.attr('date'),
  modified: DS.attr('date'),
  actor: DS.attr('string'),
  verb: DS.attr('string'),
  published: DS.attr('date'),
  metadata: DS.attr('string')
});

Create a custom RESTAdapter

The Ember-Data REST adapter expects a REST API to follow some common conventions for URL patterns and for JSON data formats. For example, if your REST API provides a collection of cats then Ember-Data will expect your REST API to work like so:

What Ember-Data expects for a cats collection:

GET /cats - get collection of cats
POST /cats - create new cat.
GET /cats/{cat-id} - get cat specified by ID.
PUT /cats/{cat-id} - update cat specified by ID.
DELETE /cats/{cat-id} - delete cat specified by ID.

Usergrid follows the above conventions for collections, but there are some exceptions. For example, the Usergrid Activities collection. A GET on the /activities path will return the Activities of the users that you (i.e. the currently authenticated user) follow. You don't POST new activities there, instead you post to your own Activities collection at the path /users/{your-user-id}/activities. It works like this:

Usergrid's Activities collection:

GET /activities - get Activities of all users that you follow.
POST /user/{user-id}/activities - create new Activity for user specified by ID
GET /user/{user-id}/activities - get Activities for one specific user.

To adapt the Activities collection to Ember-Data, I decided to create a new model called NewActivity. A NewActivity represents the data needed to create a new Activity, here's the model:

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L132" >Link)

// Must have a special model for new activity because new 
// Activities must be posted to the path /{org}/{app}/users/activities, 
// instead of the path /{org}/{app}/activities as Ember-Data expects.
App.NewActivity = DS.Model.extend({
  content: DS.attr('string'),
  location: DS.attr('string'),
  actor: DS.attr('string'),
  verb: DS.attr('string')
});

Then, in Checkin's custom REST adapter, I added logic to the pathForType() function to ensure that NewActivities are posted to the correct path. Here's the adapter:

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L40" >Link)

App.ApplicationAdapter = DS.RESTAdapter.extend({

  host: Usergrid.getAppUrl(),

  headers: function() { 
    if ( localStorage.getItem("access_token") ) {
      return { "Authorization": "Bearer " 
          + localStorage.getItem("access_token") }; 
    } 
    return {};
  }.property().volatile(), // ensure value not cached

  pathForType: function(type) {
    var ret = Ember.String.camelize(type);
    ret = Ember.String.pluralize(ret);

    if ( ret == "newActivities" ) {
      // Must have a special logic here for new activity 
      // because new Activities must be posted to the 
      // path /{org}/{app}/users/activities, instead of the 
      // path /{org}/{app}/activities as Ember-Data expects.
      ret = "/users/" + Usergrid.user.username + "/activities";
    }
    return ret;
  }

});

You can see a couple of other interesting things in the example above. First, there's the host field which specifies the base-URL of the REST API for the Checkin app. Next, there's the headers() function, which ensures that every request carries the access_token that was acquired during login.

Create a custom RESTSerializer

Ember-Data also has expectations about the JSON format returned by a REST API. Unfortunately, what Ember-Data expects and what Usergrid provides are quite different. The two examples below illustrate the differences:

Ember-Data vs. Usergrid JSON formats

Ember-Data expects collections like this:

{
   cats: [{
       "id": "6b2360d0",
       "name": "enzo",
       "color": "orange"
   },{
       "id": "a01dfaa0",
       "name": "bertha",
       "color": "tabby"
   }]
}

Usergrid returns collections like this:

{
   action: "get",
   path: "/cats",
   count: 2,
   entities: [{
       "uuid": "6b2360d0",
       "type": "cat",
       "name": "enzo",
       "color": "orange"
   },{
       "uuid": "a01dfaa1",
       "type": "cat",
       "name": "bertha",
       "color": "tabby"
   }]
}

Ember-Data expects individual objects like this:

 
{
   cat: {
       "id": "a01dfaa0",
       "name": "bertha",
       "color": "tabby"
   }
}

Usergrid returns individual objects like this:

 
{
   "id": "a01dfaa0",
   "type": "cat",
   "name": "bertha",
   "color": "tabby"
}

You can see two differences above. Ember-Data expects JSON objects to be returned with a "type key" which you can see above: the "cats" field in the collection and the "cat" field in the individual object. Also, Ember-Data expects an object's ID field to be named "id" but Usergrid returns it as "uuid."

The deal with these differences, the Checkin app extends Ember-Data's DS.RESTSerializer. Here's the code:

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L75" >Link)

App.ApplicationSerializer = DS.RESTSerializer.extend({

  // Extract Ember-Data array from Usergrid response
  extractArray: function(store, type, payload) {

    // Difference: Usergrid does not return wrapper object with 
    // type-key. So here we grab the Usergrid Entities and stick 
    // them under a type-key
    var typeKey = payload.path.substring(1);
    payload[ typeKey ] = payload.entities;

    // Difference: Usergrid returns ID in 'uuid' field, Ember-Data 
    // expects 'id'. So here we add an 'id' field for each Entity, 
    // with its 'uuid' value.
    for ( var i in payload.entities ) {
      if ( payload.entities[i] && payload.entities[i].uuid ) {
        payload.entities[i].id = payload.entities[i].uuid;
      }
    }
    return this._super(store, type, payload);
  },

  // Serialize Ember-Data object to Usergrid compatible JSON format
  serializeIntoHash: function( hash, type, record, options ) {

    // Usergrid does not expect a type-key
    record.eachAttribute(function( name, meta ) {
      hash[name] = record.get(name);
    });

    return hash;
  }
});

In the code above you can see how the extractArray() method moves the "entities" collection returned by Usergrid into a type-key field as expected by Ember-Data and how it copies the "uuid" field to add the "id" field that Ember-Data expects.

We also need to transform the data that Ember-Data sends to Usergrid. You can see this above in the serializeInHash() function, which ensures that when data is POSTed or PUT to Usergrid, the type key is removed because that's what Usergrid expects.

Implementing Add-Checkin

To implement Add-Checkin, I added an HTML template called "add-checkin" to Checkin's index.html file. The template displays an Add-Checkin form with two fields: one for content and one for the location. Here's what it looks like in all its modal glory:

Both fields are simple strings (someday I'd like to extend Checkin to use location information from the browser). I won't go into detail here, but it took a bit of research to figure out how to make a Bootstrap modal dialog work with Ember.js. Below you can see the add-checkin controller, which provides a save() function to save a new checkin.

From app.js (<a href="https://github.com/snoopdave/usergrid-ember/blob/v2/js/app.js#L390" >Link)

App.AddCheckinModalController = Ember.ObjectController.extend({

  actions: {

    save: function( inputs ) {

      var content = inputs.content;
      var location = inputs.location;
      var target = this.get("target");

      var activity = this.store.createRecord( "NewActivity", {
        content: content,
        location: location,
        verb: "checkin",
        actor: {
          username: Usergrid.user.username
        }
      });

      activity.save().then(
        function( success ) { 
          alert("Saved"); 
        },
        function( error ) { 
          alert("Error " + error.responseJSON.error_description); 
        }
      ); 

    } 
  }
});

In the code above you can see how easy it is to access Usergrid data via Ember-Data now that we've got our custom REST adapter and serializer in place. We create a new Activity with a call to this.store.createRecord() and to save it all we need to do is activity.save().

Time to wrap up...

To sum things up, here are some closing thoughts and observations.

If you are considering JavaScript MVC frameworks, then Ember.js is definitely worthy of your consideration. The documentation makes it easy to learn and the community is friendly and helpful.
It would be great for Usergrid to provide an Ember.js SDK that makes it really easy to build apps with Ember.js and Usergrid.
Ember-Data is an integral part of Ember.js, something that you need to do pretty much anything, but it is treated as a separate package with separate documentation. That is somewhat confusing for a new user.
Ember-Data does not include built-in form validation so if your app includes a large number of non-trivial forms, then you may prefer AngularJS over Ember.js.
There is a form validation plugin for Ember.js, but it requires the experimental Ember-CLI utility. I tried to use it, but Ember-CLI was unpleasnt enough that I gave up.

I appreciate any feedback you might have about this article, the Usergrid-Ember project and Apache Usergrid. If you want to see how the whole Usergrid-Ember project fits together, find it on GitHub here: Usergrid-Ember. Next up, I'll write about my experiences using Apache Shiro to replace Spring Security in Apache Roller.

Usergrid and Ember.js - part 1

Dave Johnson — Tue, 20 Jan 2015 09:39:47 +0000

The next one of my 2014 Side projects that I’d like to share is Usergrid-Ember, an experiment and attempt to learn more about Ember.js and Apache Usergrid by implementing the Checkin example from my Usergrid mobile development talk. If you're interested in either Usergrid or JavaScript web development then I hope you'll read on...

Why Ember.js?

Ember.js is one of the leading frameworks for building browser-based apps. It's one of many JavaScript Model View Controller (MVC) frameworks. Generally speaking, these frameworks let you define a set of routes or paths in your app, for example /index, /orders, /about, etc. and map each to some JavaScript code and HTML templates. Handling a route usually means using Ajax to grab some “model” data from a server and using a template to create an HTML “view” of the data that calls functions provided in a "controller" object.

JavaScript MVC frameworks are not simple and each has its own learning curve. Is it really worth the learning time when you can do so much with a little library like jQuery? For most projects I think the answer is yes. These frameworks force you to organize your code in a logical and consistent way, which is really important as projects grow larger, and they provide features that may save you a lot of development time.

Based on what I've seen on the net and local meet-ups, the leading frameworks these days are Ember.js and AngularJS. After I saw Yehudi Katz’s talk at All Things Open, I decided to spend some time learning Ember.js.

Getting started with Ember.js

The first thing you see when you visit the Ember.js site is a big button that says "DOWNLOAD THE STARTER KIT" and so that is where I started. The Starter Kit is a, a minimal Ember.js project with about twenty JavaScript, HTML and CSS files. It's a good way to start: small and simple.

Ember.js Starter Kit files:

Sidebar: I do hope they keep the Starter Kit around when the new Ember-CLI tool matures. Ember-CLI generates too many magic boiler-plate files and sub-directories for somebody who is trying to understand the basics of the framework. And this is an interesting point of view: Ember-CLI is Making You Stupid by Yoni Yechezkel.

Other stuff: Bower, Grunt and Bootstrap

I like to bite off more than I can chew, so I decided to use a couple of other tools. I used Bower to manage dependencies and Grunt to concatenate and minify those dependencies, and other things like launching a simple web server for development purposes. I also decided to use Bootstrap to provide various UI components needed, like a navbar and nicely styled list views.

I won't cover the details, but it was relatively easy to get Bower and Grunt working. Here are the config files in case you are interested: bower.json and Gruntfile.js. I did hit one problem: when I included Bootstrap as one of my dependencies the Glyphicons would all appear as tiny boxes, so I decided to pull Bootstrap from a CDN instead (looks like there is a fix for that now).

Defining Index Route, Model and Template

Every Ember.js app needs to define some routes. There is a default route for the "/" path which is called the index route, and you can add your own routes using the Router object. The snippet below shows what I needed to get started:

Part of app.js (link)

// create the ember app object
App = Ember.Application.create();

// define routes
App.Router.map(function() {
    this.route("login", { path: "/login" });  
    this.route("logout", { path: "/logout" });
    this.route("register", { path: "/register" });
});

Ember.js will look for the JavaScript Route and Controller objects as well as the HTML template using the names above. For example: Ember.js will expect the login route to be named App.LoginRoute, the controller to be named App.LoginController and the template to be named "login."

Let's talk about the index route. When a user arrives at your app they’ll be directed to the index route. Ember.js will then look for a JavaScript object called App.IndexRoute to provide the model data and JavaScript functions needed for the index page. Here’s a partial view of the index route:

Part of app.js (link)

App.IndexRoute = Ember.Route.extend( {

    // provide model data needed for index template
    model: function() {
        if ( this.loggedIn() ) {
            return this.store.find("activity");
        }
        return [];
    }

});

The index page of the Checkin app shows the Checkin activities of the people that you follow. Above you can see how to route's model() function makes that data available to the template for display. If the user is logged in we call the store.find(“activity”) function to call the Usergrid REST API to get an array of the latest Activity objects. There is some serious Ember-Data magic going on there and I'll cover that in part two of this article.

To display the index route, Ember looks for an HTML template called “index” and will use that template to display the index page. Below is the index template. The template is a Handlebars template and the things that appear in double curly-braces are Handlebars expressions.

Part of index.html (link)



  
    

      
        Checkin
      

      
          {{action 'showModal' 'add-checkin-modal' model }}>Add Checkin
      

      
        Logout 
      

    
  

  
    

      
        {{#each item in model}}
          
              {{item.content}} | {{item.location}}
          
        {{/each}}

In the above template you can see a couple of {{action}} expressions that call out to JavaScript methods defined in the Checkin app. The part of the code that uses the model is in the {{#each}} loop which loops through each Activity in the model and dispays an HTML list with the the item.content and item.location of each Activity.

Here's what the above template looks like when displayed in a browser:

Implementing Login

In Checkin, login is implemented using HTML Local Storage. Once a user has successfully logged in, the app stores the username and the user's access_token in Local Storage. When user arrives at the index page, we check Local Storage to see if that user is logged in and if not, we direct them to the login route, which in turn displays the login page using the template below.

Part of index.html (link)



  
    
      
        Checkin
      
    
  

  
    
      

        

         Please Login

         Email address
         {{input class="form-control" type="text" 
              valueBinding="username" placeholder="Username"}}

         Password
         {{input class="form-control" type="password" 
             valueBinding="password" placeholder="Password"}}
           
         
            Login
         

         Register as new user.

The LoginController provides the functions needed by the Login page itself and there are two. There is a login() function (called on line 27 above) that performs the login, and there is a register() function (called on line 31 above) that directs the user to the New User Registration page. Here's a snippet of code from the App.LoginController that provides these two functions:

Part of app.js (link)

App.LoginController = Ember.Controller.extend({

  actions: {

    login: function() { 

      // login by POST to Usergrid app's /token end-point

      var loginData = {
        grant_type: "password",
        username: this.get("username"),
        password: this.get("password")
      };

      $.ajax({
        type: "POST",
        url: Usergrid.getAppUrl() + "/token",
        data: loginData,
        context: this,
        error: function( data ) {

          // login failed, show error message
          alert( data.responseJSON.error_description );
        },
        success: function( data ) { 

          // store access_token in local storage
          Usergrid.user = data.user;
          localStorage.setItem("username", loginData.username );
          localStorage.setItem("access_token", data.access_token );

          // clear the form
          this.set("username", ""); 
          this.set("password", "");

         // call route to handle post-login transition
          this.get("target").send("onLogin"); 
        }
      });
    },

    register: function() {
      this.transitionToRoute("register");
    }

  }
});

The above code shows how to login to a Usergrid app using jQuery's Ajax feature. The login() function takes the username and password values from the login form, puts those in a JSON object with grant_type "password" and posts that object to the /token end-point of the Usergrid app. If that post succeeds, the response will include an access_token. We store that in Local Storage; we'll need to use it in all subsequent calls to Usergrid.

Usergrid fans will notice that I'm not using the Usergrid JavaScript SDK. That's because Ember.js provides Ember-Data, which acts as a very nice REST client and can be adapted to work with the URL structure and JSON formats of just about any REST API. I'll write about that in part two of this article.

2014 side projects

Dave Johnson — Mon, 5 Jan 2015 08:37:04 +0000

For various reasons, I've always got a couple of coding projects on the back burner, things that I hack around with on weekends and breaks. In 2014, I started four projects and learned about Ember.js, jQuery Mobile, Apache Shiro, Apache CXF and the Arquillian test framework.

I like to share my code, so I've put my code on GitHub and I'm going to write a brief post about each here on my blog. I'll provide links as I go and, of course, I welcome any criticisms and suggestions for improvement that you might have. First up: the Usergrid-Mobile project.

The Usergrid-Mobile project

To be honest, Budapest was the goal of this project. In the Spring of 2014, I decided that the best chance of getting to ApacheCon EU in Budapest was to create a great "mobile development with Usergrid" talk, and to do that I needed a great example project. The resulting project shows how to create a dumbed-down Foursquare-style "checkin" app using HTML5, JavaScript, jQuery Mobile and Apache Cordova.

Luckily for me, my talk was accepted for ApacheCon EU and in November I traveled to Budapest (took some photos) and gave the talk there.

I also presented the talk at the All Things Open conference in Raleigh, NC and you can view a video of that talk, Mobile Development with Usergrid on YouTube.

You can find the code for usergrid-mobile on GitHub. I also created a Vagrant File to launch a local instance of Usergrid for demo purposes. It's called usergrid-vagrant.

That's all for now. Next up: Usergrid-Ember.

Phoenix Websites

Dave Johnson — Mon, 1 Dec 2014 07:07:25 +0000

My eldest son Alex and his friend Austin have started a website design and creation business called Phoenix Websites and, of course, I think this is a great thing. They're not yet out of high school and just getting started, but they've already landed a couple of real live customers. They've got some skills and are not afraid of hard work, so if you're a Triangle-area small business owner and you need a nice new website, check them out.

Like any new business, they need some link love so here we go: Phoenix Websites: Website design and creation services in the Raleigh-Durham Triangle-area. Follow them on twitter at @phoenixrdu

Introduction to Apache Usergrid

Dave Johnson — Sat, 29 Nov 2014 10:09:48 +0000

I travelled to Budapest, Hungary for a couple of weeks for a very nice vacation with my wife and to speak at ApacheCon EU. Here are the slides that I presented at ApacheCon EU:

(you can also view the presentation on Slideshare.)

And here is the session abstract:

Whether you are building a mobile app or a web app, Apache Usergrid (incubating) can provide you with a complete backend that supports authentication, persistence and social features like activities and followers all via a comprehensive REST API — and backed by Cassandra, giving you linear scalability. All that, and Usergrid is open source too.

This session will explain how you can use Usergrid to provide a back-end for your application. We’ll start with an overview of Usergrid features, then explore in depth how to authenticate users, store data and query data with the REST API provided by a Usergrid server. We’ll develop a simple HTML5 app and package it as a native mobile app via Apache Cordova. We'll also cover how to run Usergrid locally for development and testing.

Talking Usergrid at ApacheCon 2014

Dave Johnson — Sun, 16 Mar 2014 16:10:16 +0000

I've been working at Apigee since September 2013 and one of the things I love most about my new job is the fact that I'm actively contributing to open source again.

I'm working on Apache Usergrid (incubating), an open source Backend-As-A-Service (BaaS) that's built on the Apache Cassandra database system. Apigee uses Usergrid as part of Apigee Edge (see the Build Apps part of the docs).

Apigee contributed code for Usergrid to the Apache Software Foundation back in October 2013 and Usergrid is now part of the Apache Incubator. The project is working towards graduating from the Incubator. That means learning the Apache way, following the processes to get a release out and most importantly, building a diverse community of contributors to build and maintain Usergrid.

One on the most important parts of building an open source community is making it easy for people to contribute and and that's why I submitted a talk to the ApacheCon US 2014 conference (April 7-9 in Denver, CO) titled How to Contribute to Usergrid.

The talk is intended to be a briefing for contributors, one that will lead you through building and running Usergrid locally, understanding the code-base and test infrastructure and how to get your code accepted into the Usergrid project.

Here's the outline I have so far:

How to Contribute to Apache Usergrid

Motivation
- Why would anybody want to contribute to Usergrid?
First steps
- The basics
- Getting signed up
Contributing to the Stack
- Understanding the architecture & code base
- Building the code. Making and testing changes
- Running Usergrid locally via launcher & via Tomcat
Contributing to the Portal
- Understanding the architecture & code base
- Building the code. Making and testing changes
- Running the portal locally via node.js
Contributing to the SDKs
- Understanding the architecture & code base
- Building the code. Making and testing changes
Contributor workflow: how to get your code into Usergrid
- For quickie drive-by code contributions
- For more substantial code contributions
- For documentation & website changes
Contributing Docs and Website changes
- Website, wiki and GitHub pages
- How to build the website and docs
Roadmap
- First release
- New Core Persistence system
- The two-dot-o branch
- Other ideas

I'm in the process of writing this talk now so suggestions and other feedback are most welcome.

New Bootstrap based theme

Dave Johnson — Sun, 19 Jan 2014 21:10:01 +0000

I'm not motivated to write new blog entries but for some reason I was motivated to update my blog's theme. This time I decided to go with Twitter Bootstrap + jQuery. It's responsive, so to speak. What do you think? I think it takes a little too long for the banner image to load.

Upgraded to latest 5.1.0-SNAPSHOT

Dave Johnson — Sun, 3 Nov 2013 11:26:07 +0000

Just a quick note about this site:

Today I upgraded this site to the latest Roller 5.1.0-SNAPSHOT (unreleased) code base. I did this to fix a couple of vulnerabilities found recently by researchers from Coverity (thanks Coverity!).

The Roller team fixed the vulnerabilities reported by Coverity in both the Roller 5.1-SNAPSHOT (unreleased) and the Roller 5.0.x branch. If you're running Roller, you should upgrade to Roller 5.0.1 immediately.

It's been while since I upgraded this site (or even blogged here) so the upgrade was a little painful. When I first upgraded I found that the JSPWiki plugin was broken and the site would not even load. Turns out, I was using an archaic copy of the JSPWiki jar and it was compiled against an equally archaic version of Lucene. Since we recently upgraded the version of Lucene included in Roller, this broke things. I fixed this by upgrading the Roller JSPWiki plugin to use the latest code from Apache JSPWiki. I also created a Github repo with the new JSPWiki code, which you can find here: Roller JSPWiki Plugin on Github.

How Wayin does Cloud Orchestration

Dave Johnson — Tue, 18 Jun 2013 16:16:34 +0000

I've already mentioned this on Twitter and LinkedIn, but just in case you missed it: I'll be speaking tomorrow night at the Triangle AWS and Triangle DevOps joint meetup at Argyle Social in Durham, NC. I'll give a quick overview of cloud orchestration and Wayin Hub. Then I'll dive into the details of how we automate deployment, scaling and backups for Wayin Hub using AWS and AWS Cloud Formation.

As a little teaser, here's a GIF animation of my automated deployment slide:

For more information check the Triangle DevOps page for the event.

UPDATE: slide are now on SlideShare.

Composite Keys in Apache Cassandra

Dave Johnson — Thu, 23 May 2013 17:56:49 +0000

Newer versions of Apache Cassandra include CQL, an SQL-like query language that supports both query, update and delete statements as well as the Data Definition Language (DDL) statements like create and alter for tables and indexes.

You can create tables (known as column families in Cassandra lingo) just like you can in a relational database, but there are some caveats. One caveat is this: if you want to sort and use ORDER BY in queries on a table, then you will have to use composite-key in that table and that composite key must include the field that you wish to sort on. You have to decide which fields you wish to sort on when you design your data model, not when you formulate queries. Another caveat is that, with Cassandra 1.1, there is no support for secondary indexes on composite-keyed tables. That means you can only query on the fields in the composite-key and in certain specific ways. More on that later.

In version 1.1, Cassandra supports (at least) two different models for storing data. You can use a single primary key in your table, or you can use a composite key. Cassandra stores your data differently for these two cases and the queries that you can perform on these two types of tables vary as well.

I’ll explain how simple-keyed and composite-keyed differ with some examples. The example is the Bite table, which holds a chunk of data identified by an ID and sorted by an integer score value. I don’t want to get into details, but in the product I'm working on, Wayin Hub, a Site contains Feeds and Feeds contain Bites, which might represent Tweets, RSS items or other social network activities. Hopefully, that will be enough background for you to understand the examples below.

Single-keyed Table

With a single keyed table you have a row for each entity that you store and a column in that row for each field of the entity. For example, to represent the Bite table as a single-keyed table it would be defined like so:


   create table bite (
	id varchar PRIMARY KEY,
	feedid varchar,
	score bigint,
	data varchar
   );

   create index bite_feedid on bite (feedid);
   create index bite_score on bite (score);

We need those id, feedid and score fields so we can look up bites by those values. The data field is used to store a JSON representation of other data we associate with each Bite. For example, if you only have three Bites in the table, here’s what the results of a select * from bite might look like:


 id   |  feedid  |  score  | data
------+----------+---------+----------------------
bite2 |  feed0   |   101   | { “status” : “APPROVED”, ... 
bite3 |  feed0   |   102   | { “status” : “DENIED”, ...
bite1 |  feed0   |   100   | { “status” : “APPROVED”, ...

The way the data is stored in Cassandra would look about the same, as illustrated in the diagram below. Each table row corresponds to a Row in Cassandra, the id of the table row is the Cassandra Row Key for the row. Each value in the row is a Cassandra Column with a key and a value. If you add more table rows, you get more Cassandra Rows.

Now let’s get back to the topic of this post and that caveat that I mentioned earlier.

Composite-keyed Table

If we want to sort data in a table, then we need to use a composite-keyed table. With a composite-keyed table you define a composite-key made up of multiple fields from the table. The first key is known as the partition-key. To sort by score we also include the score in the composite key. And since it is possible for two Bites with the same partition key to occur at the very same time, we also include varchar ID to ensure uniqueness.

Here’s how the Bite table is defined. The composite-key is the list of three fields in PRIMARY KEY parentheses.


   create table Bite (
	 partkey varchar,
	 score bigint,
	 id varchar,
	 data varchar,
	 PRIMARY KEY (partkey, score, id)
   ) with clustering order by (score desc);

And if you had three Bites in the table the query select * from bite would return this:


 partkey  |  score  |  id   |  data
----------+---------+-------+----------------------
  feed0   |   102   | bite3 | { “id” : “bite2”, ... 
  feed0   |   101   | bite2 | { “id” : “bite3”, ...
  feed0   |   100   | bite1 | { “id” : “bite1”, ...

The surprise is how this table is stored in Cassandra. Instead of storing a Cassandra Row for each table row, the data is stored as one row. The more Bites you add to the table, the more Cassandra Columns are added to that Row. The diagram below illustrates how this works for the three Bites of data above. There is one Row with key of feed0. And there is one Column for each table row of data. Each Column uses a key that combines the score and id plus a string that indicates what field is stored in the Column Value, which in our case is only the data field.

Querying a Composite-keyed Table

With Cassandra 1.1, if you want to select a single Bite, you must know all of the composite-keys. Here’s an example query that selects a single Bite:


   select data from bite where partkey=’feed0’ and score=101 and id=’bite2’

To get latest Bites in a Site’s Feed, you specify only the partition-key and ask for ordering by score, like so:


   select data from bite where partkey=’feed0’ order by score desc limit 20

If you try to query without specifying the partition key and the score, you will get an error message. For example, this query:


   select data from bite where id=’bite2’ limit 20

Would give you this error message:


   Bad Request: PRIMARY KEY part id cannot be restricted (preceding
   part score is either not restricted or by a non-EQ relation)

That means we can’t look up Bites by a single ID. That’s not very convenient but that’s the way it is with Cassandra 1.1 which does not allow additional indexes on composite-key tables.. If you really want to lookup Bites by id, you have to create an entirely new simple-keyed table with Bite id as the primary key and use that table to look up the a Bite’s partKey and score. This problem is fixed in Cassandra 1.2 because it allows secondary indexes on fields in composite-key table. Let’s talk about that.

Secondary Indexes

Cassandra 1.2 comes with support for secondary indexes on composite-keyed tables, but you cannot create a secondary index on keys that are already part of the composite-key. So, if we want to be able to look-up Bites by ID, then we must add a second and redundant biteid field like so:


   alter table Bite add biteId varchar;
   create index Bite_biteId on Bite (biteId);

Inside Cassandra, the the data would look like this, a new field in the table means a new Column in the row, as show below:

And with that secondary index we can support queries like this:


   select data from bite where biteid=’bite2’

Cassandra 1.2

In the DataStax Cassandra 1.2 docs, it says “CQL3 transposes data partitions (sometimes called "wide rows") into familiar row-based result sets, dramatically simplifying data modeling.

I believe that means that all tables are stored the way that composite-keyed tables are stored. According to the docs, legacy tables from Cassandra 1.1 are supported in 1.2 and, if you want, you can still create Cassandra 1.1 style tables by using the “compact storage” attribute. For example, to create the Bite table with a the Cassandra 1.1 table model and a single primary key you’d do this:


   create table bite (
	id varchar PRIMARY KEY,
	feedid varchar,
	score bigint,
	data varchar
   ) with compact storage;

And that’s all I’ve got on this topic.

Wrapping up...

I wrote this up to help myself understand how composite-keyed tables work in Cassandra, so I’d love any feedback you might have and especially if you think I’ve got concepts or terminology wrong. Thanks for reading.

You can read more about Cassandra 1.1 tables on the Datastax site:
http://www.datastax.com/docs/1.1/ddl/column_family

More about Cassandra 1.2 tables:
http://www.datastax.com/docs/1.2/ddl/table

More about Cassandra 1.1 legacy tables in Cassandra 1.2
http://www.datastax.com/docs/1.2/ddl/legacy_table

Also, I found these posts by Brian O’Neill very helpful:
http://brianoneill.blogspot.com/2012/09/composite-keys-connecting-dots-between.html
http://brianoneill.blogspot.com/2012/10/cql-astyanax-and-compoundcomposite-keys.html

And, if you want to build an engaging site for your customers, fans or constituents based on live tweets, photos and videos check out Wayin Hub and follow @wayinhub on Twitter.

WIP #6: Feed-based Integration

Dave Johnson — Mon, 8 Oct 2012 10:38:43 +0000

This is the sixth in my series of Web Integration Patterns. Check out the intro at this URL http://rollerweblogger.org/roller/entry/web_integration_patterns

Synopsys

This pattern is about integrating web sites and applications by using standard feed formats to convey timely information, updates, status messages, events and other things from one web application to another.

Motivations

Take advantage of the many tools and services that can produce, consume, aggregate and build work-flows around feeds as a way to integrate web applications.
Use a standards-based approach to ensure that that widest range of feed producers and consumers can be supported. Make it so everybody who needs can read your feeds.

Related Patterns

Search Integration
Web APIs

A feed is a list of entries, each with an timestamp, ID, title, content and metadata like categories and tags. Entries are arranged in reverse chronological order. The entries in a feed can represent just about anything from blog entries, Flickr photos, YouTube videos, source-code change sets or tasks in a change management system. A feed is an XML resource that is available at a URL. If you want updates, then you poll that URL, ideally using HTTP Conditional GET so that you only pull down the feed when it has been updated.

Generally speaking there are two standard feed formats in use on the web: RSS and Atom, both are based on XML. Both use different element names, for example: what Atom calls "entries" RSS calls "items." Because these standard formats are so widely supported, providing a feed is an effective way to share updates from your web site or application.

Elements of Atom (from my 2006 presentation on Atom)

Another flavor of feeds is ActivityStrea.ms, which is essentially a feed format with a schema for representing about 70 different types of activities. These activities can be social network activities like share or friend and they can also be business activities like assign, resolve or schedule. One advantage of using the ActivityStrea.ms standard is that it has both an Atom and a JSON mapping.

Producing Feeds

Feed-based Integration is listed as a basic pattern because it can be very easy to implement. The ability to produce feeds is built-in to many different types of web applications from blog and wikis to continuous integration servers. If you are writing your own web application, you can choose to use XML tools to produce your feeds, a templating engine or a dedicated feed toolkit like ROME. You’ll find plenty of XML tools and templating engines no matter what language you are using.

Which type of feed should you produce? That depends. Atom is the most complete specification and is a true IETF standard, so often it is the right choice. To make the right decision, you have to consider who is going to be consuming your feeds. If your consumers prefer RSS, them give them that. If your consumers prefer JSON over XML, then consider ActivityStrea.ms in JSON flavor.

Consuming Feeds

You can use a wide variety of tools to parse and process feed data. For example, there are many web sites and services that can digest feeds and trigger other events and processing. Services such as Yahoo Pipes and If This Then That can read feeds, process each item and perform other actions based on item values.

Processing a feed with Yahoo Pipes (from Marsh Gardiner's post)

You’ll also find a variety of JavaScript widgets that can parse and display feeds in a website. Most web portal and software with a web dashboard features will include a component that can display feeds. And, of course, you can use a reader that supports RSS/Atom feeds, such as Google Reader or Feedly or FlipBoard, to monitor and read feeds.

If you need to add RSS/Atom reading features to your own software, you can use standard XML parsing tools and, for most language, you’ll find that there are open source libraries specifically designed for parsing feeds.

For more information in producing and consuming feeds, RSS and Atom in Action. Also, you might find my presentation on Beyond Blogging: Atom Format and Protocol useful.

Beyond the basics

Feeds are a great way to do simple integrations, but there are limitations and there will be times you’ll need to go beyond the basics with RSS and Atom feeds. Here's an example. Normally, with feeds, clients have to repeatedly poll the feed URL for updates. This is annoying, and inefficient, even with HTTP Conditional GET. To address this problem, you can setup a PubSubHubub server that will subscribe to feeds and will then notify other subscribers instantly when updates are available, so that those other subscribers don’t have to poll.

Another problem is that, if you don’t poll often enough, you might miss some updates and they may “scroll” off the bottom of the feed before you see them. Feed providers can address this problem by supporting Feed Paging and Archiving, which allows clients to use next and previous links to “page” back to feed items that are no longer in the first page of the feed.

One more beyond-the-basics item to mention is the related pattern Web APIs, which we'll cover later. Web APIs are listed as a related pattern to Feed-based Integration. That's because feeds have been used as the basis for several "Web APIs" or protocols. These protocols specify how to use HTTP POST, GET, PUT and DELETE and create, retrieve, update and delete web resources that are represented as feed entries. Examples are the IETF's Atom Protocol, Microsoft's OData and Google's GData APIs.

That’s it for Feed-based Integration. In my next posts, we'll move in to the Advanced Patterns.

10 years ago today

Dave Johnson — Tue, 17 Apr 2012 07:41:07 +0000

Ten years ago on this day, O'Reilly published an article that I wrote called Building an Open Source J2EE Weblogger, the article that introduced the Roller weblogger (now known as Apache Roller) to the world. It changed my career and my life in a bunch of nice ways and 10 years later I'm still benefiting from my choice to create Roller and write that article. So you can get a taste of the times, here's the intro:

Building an Open Source J2EE Weblogger: As a Java developer, you should be aware of the tremendous wealth of open source development software that is available for your use -- even if you have no desire to release any of your own software as open source. In this article, I will introduce you to some of the most useful open source Java development tools by showing you how I used these tools to develop a complete database-driven Web application called Roller.

Roller fits into the relatively new category of software called webloggers: applications that make it easy for you to maintain a weblog, also known as a blog -- a public diary where you link to recent reading on the Web and comment on items of interest to you.

The Roller Web application allows you to maintain a Web site that consists of a weblog, an organized collection of favorite Web bookmarks, and a collection of favorite news feeds. You can define Web pages to display your weblog, bookmarks, and news feeds. By editing the HTML templates that define these pages, you have almost total control over the layout and appearance of these pages. Most importantly, you can do all of this without leaving the Roller Web application -- no programming is required.

I've written and talked about Roller and the history of Roller numerous times. If you're interested in learning more about it here's my most recent Roller presentation, which covers Roller history in some detail:

Whats New In Roller5

View more presentations from snoopdave

These days, Roller isn't really thriving as an open source project. Wordpress became the de facto standard blogging package and then micro-blogging took over the world. There are only a couple of active committers and most recent contributions have come via student contributions. Though IBM, Oracle and other companies still use it heavily, they do not contribute back to the project. If you're interested in contributing to Roller or becoming part of the Apache Software Foundation, then Roller needs YOU!.

Apache Rave: A new social mashup engine!

Dave Johnson — Wed, 28 Mar 2012 08:54:00 +0000

Congrats to the Apache Rave team on reaching top-level project status.

OpenSocial API Blog: Announcing Apache Rave: The project started only a year ago, March 1 2011, when entering the Apache Incubator as a collaborative effort by individuals from a wide range of corporations, non-commercial organizations, and institutes from around the world and was seeded by code donations from The MITRE Corporation, Indiana University Pervasive Technology Institute, SURFnet, OSS Watch, Hippo, and numerous individual developers.

Rave builds on open standards and leverages and aligns with other open source projects like Apache Shindig and Apache Wookie to deliver a lightweight, flexible, widget-based platform that easily scales across federated application integrations, social intranets, and multi-channel social communities with enhanced personalization and customized content delivery.

Amazon cloud: half-a-million Linux servers

Dave Johnson — Sat, 17 Mar 2012 07:18:25 +0000

Interesting speculation on the make-up of Amazon's cloud:

Amazon EC2 cloud is made up of almost half-a-million Linux servers | ZDNet :

We know that Linux on servers is big and getting bigger. We also knew that Linux, thanks to open-source cloud programs like Eucalyptus and OpenStack, was growing fast on clouds. What he hadn’t know that Amazon’s Elastic Compute Cloud (EC2), had close to half-a-million servers already running on a Red Hat Linux variant.

Huang Liu, a Research Manager with Accenture Technology Lab with a Ph.D. in Electrical Engineering whose has done extensive work on cloud-computing, analyzed EC2’s infrastructure and found that Amazon EC2 is currently made up of 454,400 servers.

JIRA finally gets its own REST API

Dave Johnson — Thu, 1 Mar 2012 09:20:30 +0000

JIRA's got a real REST API now:

REST easy with JIRA 5 | Atlassian Blogs: Now that JIRA 5 is out, let’s talk about one of my favorite features of this new release, JIRA’s new REST API. JIRA has supported remote APIs for many years with SOAP, XML-RPC, and JSON-RPC. However, telling developers that you support SOAP (and only SOAP) is like saying that you like writing applications with COBOL — it’s out of style. Today’s cool kids gravitate towards REST. It’s clean, simple, and highly portable across languages and frameworks.

And checkout the nice looking API docs, which look like they were generated by WADL-to-HTML.

An alternative to Atlassian's new API is the recently release Rational OSLC Adapter for JIRA, which allows you to do more sophisticated integrations with JIRA including delegated UIs for issue creation and selection.

WIP: Widgets and Gadgets

Dave Johnson — Wed, 22 Feb 2012 07:52:11 +0000

This is the fifth in my series of Web Integration Patterns. Check out the intro at this URL http://rollerweblogger.org/roller/entry/web_integration_patterns

Synopsis

Allow other web sites and applications to integrate your site into their web pages by providing an embeddable user interface, commonly known as a Gadget or Widget, which allows users to view and interact with your site in the context of other sites.

Motivations

By embedding Widgets in your site, you can make your site more useful and informative to your users. Users can access relevant information from other sites in the context of your web site.
By allow other sites to embed your Widgets, you can give your site and the services that it offers wider reach. Your users can access and interact with your services in the context of other sites.

Related Patterns

Dashboards and Portals
Social Network Plugins
Delegated Pickers

A Widget or Gadget is a piece of an application, a visual component or a chunk of user-interface that can be added to some other application. On the web, Widgets are implemented using HTML and JavaScript, or sometimes Flash. Here we'll discuss simple Widgets and leave discussion or more advanced usage of Widgets in Dashboards, Portals and Pickers for the more advanced patterns we'll discuss later.

Easy to "consume"

Widgets are easy, especially for widget "consumers" i.e. those who wish to add an existing Widget to their site. Adding a widget to your site or web application can be as easy as adding a couple of lines of HTML and JavaScript to the pages of your site. You can see some examples of Widgets in the two screen shots I've included here, below is the Wayin polling Widget, which shows my latest Wayins:

And below is the Flickr Widget, which shows a random selection of my photos from Flickr. You can see these two Widgets in action on the front page of my blog. And, by the way, the Like, Tweet and +1 buttons on the right of this entry (when viewed on my blog site) are also Widgets.

Relatively easy to create

Developing an entirely new Widget more complicated than using one that already exists. Creating a new Widget typically requires writing HTML and JavaScript code and employing some interesting tricks. The simplest way to implement a Widget is to use an iframe, allowing a portion of one site to be embedded in another site.

I'm not going to go into detail as there are already so many good resources on the web for learning about Widget crafting. For example, Alex Marandon wrote a pretty comprehensive blog post on creating Widgets covering basic issues like how to ensure that your widget code does not interfere with the page hosting the Widget, how to dynamically load resources and how to bypass browsers’ single-origin policy using JSON-P.

Widget and Gadget standards

For simple Widgets you don't really need any standards other than HTML and JavaScript. If you want to target specific web site Dashboards or Portals, like NetVibes, or enterprise web applications like Jive or Atlassian's software development tools, then you'll need to create your Widgets with specific APIs in mind. For example, NetVibes provides it's own Widget API, while Jive and Atlassian support the Open Social Gadget APIs. I'll talk more about these issues when I cover the related patterns Dashboards, Portals and Social Network Plugins.

Wrapping up...

That's it for Widgets and Gadgets. Next up: Feed-based Integration.

Twitter moving away from hashbang URLs

Dave Johnson — Tue, 21 Feb 2012 13:02:28 +0000

This is such good news for the web that it gets its own blog entry. According to Dan Web, an engineer at Twitter:

@timhaines plus, now I'm in charge of undoing twitters hashbang URLs I can 

    confirm that all the issues in that article are very real.

@danwrong x months from now (when your project completes) Twitter will no 

    longer use hashbangs?

@timhaines correct. All gone. It was a mistake for several reasons. 

    PushState or bust.

The whole conversation is here:
http://storify.com/timhaines/hashbang-conversation

Here's an excellent chapter on HTML 5 history and PushState: http://diveintohtml5.info/history.html

And in case you've forgotten, here's why hashbangs suck.

The meaning of the Facebook like button

Dave Johnson — Sat, 11 Feb 2012 16:42:54 +0000

far as I can tell, clicking theFacebook "Like" button means "please spam me endlessly"
— Dave Johnson (@snoopdave) February 5, 2012

HTML5 Microdata and Schema.org

Dave Johnson — Thu, 9 Feb 2012 08:48:06 +0000

Worth a read and related to my Web Integration Patterns post on Embedded Properties in HTML:

HTML5 Microdata and Schema.org:HTML5 Microdata and Schema.org On June 2, 2011, Bing, Google, and Yahoo! announced the joint effort Schema.org. When the big search engines talk, Web site authors listen. This article is an introduction to Microdata and Schema.org. The first section describes what HTML5, Microdata and Schema.org are, and the problems they have been designed to solve. With this foundation in place section 2 provides a practical tutorial of how to use Microdata and Schema.org using a real life example from the cultural heritage sector. Along the way some tools for implementers will also be introduced. Issues with applying these technologies to cultural heritage materials will crop up along with opportunities to improve the situation.

Understanding Google's new privacy policy

Dave Johnson — Thu, 2 Feb 2012 13:39:11 +0000

Worth a read:

Understanding Google's new privacy policy: When Google changed its privacy policy last week, they made a strong effort to ensure that everyone knew that a change had occurred, but if you tried to figure out what had actually changed, you had to wade through a lot of buzzwords and legalese. Now the Electronic Frontier Foundation's Rainey Reitman explains it in simple language.

How to select an Open Source project

Dave Johnson — Tue, 31 Jan 2012 07:00:15 +0000

A reasonable check-list for selecting an open source project:

How I select Open Source projects: An Open Source project is more than its set of tools. But this post is not about what is better, github or ASF. Much has been said already about it (and too much bullshit). What really bugs me is that people seem to choose Open Source projects after the tools the projects use. Here is my personal list after which I choose projects.

IBM abandons Lotus Symphony

Dave Johnson — Tue, 31 Jan 2012 06:53:56 +0000

Apparently 2001 called and asked for its office suite back.

IBM to close down Symphony, its OpenOffice fork | ZDNet : This move can’t come as too much of a surprise. In July 2011, IBM started donating its Symphony code to the Apache Foundation, which is now overseeing the remains of the OpenOffice project. I say “remains” because most of the development energy in the OpenOffice family has been going to the LibreOffice fork. Many users and Linux distributors, such as Ubuntu, have replaced OpenOffice with LibreOffice.

Scott McNealy's Wayin weighs in on the election

Dave Johnson — Thu, 26 Jan 2012 08:12:58 +0000

Nice coverage of Wayin in CNN Money:

Scott McNealy's Wayin weighs in on the election - Fortune Tech: Nearly 30 years after he co-founded Sun, and two years after the company sold to Oracle (ORCL), one of the tech industry's most outspoken and colorful executives is back. Only instead of selling servers and software to businesses, he's trying to get consumers, sports teams, and corporations -- and politicians -- to embrace a social media tool that's a little hard to define: It is part polling app, part interactive media tool (think Twitter), and part enterprise service.

The polling software at the heart of Wayin is super easy to use. It allows you to create basic questionnaires, on the web or using a mobile app, by selecting an image and asking a question. Pick a photo of the Fab Four, ask Who is your favorite Beatle?, and bingo, the answers from your followers will start trickling in immediately.

Tim Bray on HttpURLConnection

Dave Johnson — Mon, 23 Jan 2012 08:26:37 +0000

Good information on a bad API. The comments are good too:

HttpURLConnection's Dark Secrets: If you’re programming in the Java language and want to talk to a Web server, there are several libraries you can choose from. HttpURLConnection is one popular choice, and for Android programming, the engineering team has now officially suggested that you use it where possible.

O'Grady on DynamoDB

Dave Johnson — Fri, 20 Jan 2012 07:57:12 +0000

Great analysis, as usual, from Stephen O'Grady of Redmonk. Here's the part about Dynamo that concerns me, the lock-in:

Amazon DynamoDB: First Look: Like the initial wave of PaaS platforms, however, DynamoDB is available only through a single provider. Unlike Amazon’s RDS, which is essentially compatible with MySQL, DynamoDB users will be unable to migrate off of the service seamlessly. The featureset can be replicated using externally available code – via those projects that were originally inspired by DynamoDB, for example – but you cannot at this time download, install and run DynamoDB locally.

Amazon DymamoDB

Dave Johnson — Thu, 19 Jan 2012 07:51:54 +0000

Worth a read:

Werner Vogels: Amazon DynamoDB is a fully managed NoSQL database service that provides fast performance at any scale. Today’s web-based applications often encounter database scaling challenges when faced with growth in users, traffic, and data. With Amazon DynamoDB, developers scaling cloud-based applications can start small with just the capacity they need and then increase the request capacity of a given table as their app grows in popularity. Their tables can also grow without limits as their users store increasing amounts of data. Behind the scenes, Amazon DynamoDB automatically spreads the data and traffic for a table over a sufficient number of servers to meet the request capacity specified by the customer. Amazon DynamoDB offers low, predictable latencies at any scale.

The joys of solitude

Dave Johnson — Mon, 16 Jan 2012 07:37:54 +0000

Worth a read:

The Rise of the New Groupthink - NYTimes.com: Solitude has long been associated with creativity and transcendence. “Without great solitude, no serious work is possible,” Picasso said. A central narrative of many religions is the seeker — Moses, Jesus, Buddha — who goes off by himself and brings profound insights back to the community.

Obama Administration Comes Out Against SOPA, PIPA

Dave Johnson — Sat, 14 Jan 2012 14:59:33 +0000

I feel pretty strongly about this. I won't vote for a politician who backs SOPA or PIPA:

Obama Administration Comes Out Against SOPA, PIPA | TPM Idea Lab:Even more promising for critics of the bills, the Administration came down firmly against one of the most vehemently opposed portions of the bills — the part that would give the government the power to force Internet Service Providers to stop loading overseas webpages accused of piracy. Under the original versions of SOPA and PIPA, ISPs would be required to change their Doman Name System settings to block sites accused of piracy, a measure that critics said would essentially break the Internet and make it more insecure.

NYC gets a Software Engineering High School

Dave Johnson — Fri, 13 Jan 2012 15:44:58 +0000

Worth a read:

New York City gets a Software Engineering High School - Joel on Software:This fall New York City will open The Academy for Software Engineering, the city’s first public high school that will actually train kids to develop software. The project has been a long time dream of Mike Zamansky, the highly-regarded CS teacher at New York’s elite Stuyvesant public high school. It was jump started when Fred Wilson, a VC at Union Square Ventures, promised to get the tech community to help with knowledge, advice, and money.

The coming war on general-purpose computing

Dave Johnson — Wed, 11 Jan 2012 16:58:15 +0000

Cory Doctorow's keynote to the Chaos Computer Congress. Worth a read:

Lockdown: The coming war on general-purpose computing - Boing Boing