5. CouchDB and Domain-Driven Design

This post covered two topics that are of great interest to me. Document databases like CouchDB have much potential when it comes to domain-driven design. The post outlined some techniques for serializing domain objects into CouchDB documents.

4. Addressing the NoSQL Criticism

I heard a lot of NoSQL bashing this year at OSCON. As a result, I decided to write a rebuttal to many of the criticisms that I heard. This post inspired a lively comment thread.

3. Testing PHP 5.4

This was a short post on how to help the PHP Quality Assurance Team by running the tests for the latest development version of PHP 5.4. It covered downloading, compiling, and running the tests. Most importantly, it also covered reporting your test results.

2. Exploring RabbitMQ and PHP

I was exploring the possibility of using RabbitMQ for an upcoming project. I decided to post some notes on my experience getting started with RabbitMQ and integrating it with PHP. I haven’t ended up using RabbitMQ in production yet, but I’m confident that I’ll end up using it at some point.

1. CouchDB jQuery Plugin Reference

I had been frustrated by the lack of documentation for the CouchDB jQuery plugin—so decided to write my own. After I wrote the blog post, Max Ogden pointed me to Dale Harvey‘s documentation generator for jquery.couch.js and a copy of the generated jQuery CouchDB documentation. Max and Dale are also working on a more flexible jquery.couch2.js.

• Embrace, and don’t abstract, the Hypertext Transfer Protocol (HTTP).
• Focus on entities/resources—identified by full Uniform Resource Identifiers (URIs).
• Use HTTP methods (GET, POST, PUT, DELETE, OPTIONS, HEAD) to perform operations on entities/resources.
• Allow for self-described messages through the use of header fields, such as Accept and Content-Type.
• Make hypermedia controls a core concept (perhaps using PHACTOR as a starting point), not just a byproduct of rendering.
• Think of web applications as state transition systems. Representations of entities/resources are states, and hypermedia controls define the available state transitions.

Benefits:

• Interoperability: Focusing on open standards allows for easier integration with other systems.
• Cacheability: Embracing HTTP gives you many caching options, almost for free.
• Testability: Self-contained and self-describing messages are very testable. Decomposing a complex system into states and available state transitions greatly reduces the complexity of the system and its tests.

I’ll be using RabbitMQ first from PHP, but I plan on using it to send and receive messages to and from other systems. Following are the steps I used to get RabbitMQ and PHP’s AMQP library setup on my development machine.

First, I installed RabbitMQ using MacPorts:

$ sudo port install rabbitmq-server

Then, I started RabbitMQ:

$ sudo rabbitmq-server -detached

Next, I installed the librabbitmq library using a slight variation of the instructions on PHP’s AMQP Installation page (you may need to install Mercurial first):

$ hg clone http://hg.rabbitmq.com/rabbitmq-c/rev/3c549bb09c16 rabbitmq-c
$ cd rabbitmq-c
$ hg clone http://hg.rabbitmq.com/rabbitmq-codegen/rev/f8b34141e6cb codegen
$ autoreconf -i && ./configure && make && sudo make install

Then, I installed the AMQP extension using PECL:

$ sudo pecl install amqp-beta

To test that everything works, I opened up two interactive PHP shells using php -a. I ran the following code in the first PHP shell:

$exchangeName = 'messages';
$routeKey = 'routeA';
$message = 'Hello, world.';
$connection = new AMQPConnection();
$connection->connect();
$exchange = new AMQPExchange($connection);
$exchange->declare($exchangeName);

I then ran the following code in the second PHP shell:

$exchangeName = 'messages';
$routeKey = 'routeA';
$connection = new AMQPConnection();
$connection->connect();
$queue = new AMQPQueue($connection);
$queue->declare($exchangeName);
$queue->bind($exchangeName, $routeKey);

Back in the first PHP shell:

$exchange->publish($message, $routeKey);

Back in the second PHP shell:

$message = $queue->get();
print_r($message);

Here is the output I got from the print_r statement:

Array
(
    [routing_key] => routeA
    [exchange] => messages
    [delivery_tag] => 1
    [Content-type] => text/plain
    [count] => 0
    [msg] => Hello, world.
)

There are several other options that can be set, and a lot more to learn about RabbitMQ and AMP. Check out the documentation for PHP’s AMQP extension for details about working with AMQP servers from PHP.

Atom Syndication Format

Most of VPR’s web content is currently stored in a proprietary content management system (CMS). The first step was publishing the needed content from the CMS out into an open format. VPR’s Dan Allen published from the CMS Atom feeds of the various content we would need for the new homepage.

PHP and Zend Framework

Once we had the needed Atom feeds, we created a Zend Framework web application to import these feeds. Components such as Zend_Feed_Reader made the job easier. The weather data comes from the National Oceanic and Atmospheric Administration’s (NOAA) National Weather Service’s National Digital Forecast Database XML/REST Service. The market data comes from the Google Finance API.

CouchDB

All of the data (except for weather, which is pulled directly with a bit of caching) is stored in CouchDB. Since we’re storing documents with varying metadata (i.e. Atom entries) choosing a document-oriented database made a lot of sense. CouchDB’s map/reduce views make for very efficient queries. Its RESTful HTTP API provides a lot of options for scaling and caching. We’re caching documents within the Zend Framework application and conditionally requesting these documents from CouchDB with ETags when a cache is available. The current cache is simply file-based but we’re considering using Memcached for even better performance. On a related note, the homepage itself also supports conditional HTTP requests.

Apache and nginx

We had the pleasure of working with another great Vermont company, ClearBearing, on this project. ClearBearing handles the infrastructure layer for VPR.net. The current website was (and still is) served up by Apache. ClearBearing already had nginx in place as a reverse proxy to handle caching. Since we weren’t replacing the entire CMS we needed a way for the existing CMS and this new application to coexist. We decided to host the new application separately (still administered by ClearBearing) and reverse proxy requests for certain URIs (primarily the homepage) to the new application. This provides a clean separation between the new application and the existing CMS while still allowing for them to be hosted on the same domain from an end-user perspective.

Licensing

You’ll notice that all of the standards used are open standards and all of the technologies used are free and open source software. Additionally, all of the code we have written for VPR is licensed to them under the permissive New BSD License. This means that VPR is free to use this code however they want, study and modify the code, and redistribute the code as-is or modified. They can do all of this without paying additional licensing fees.

Design

While this post is primarily about the technology behind VPR’s new homepage, I should mention that Found Line’s Jason Pelletier designed the new homepage and implemented all of its HTML and CSS. We worked closely with Jonathan Butler and John Van Hoesen, Vice President for News & Programming, to make sure the new design meets the needs of website visitors and the organization.

Next Steps

The launch of this new homepage was just one iteration with many more to come. As Jonathan Butler mentioned in his blog post, VPR has plans to further improve online services for their listeners. Look for a redesigned VPR Classical page soon. There are also plans for better mobile access to VPR and VPR Classical programming. Fortunately much of the work that has been done can be leveraged to create additional online services for VPR’s listeners.

A special thanks for the support of VPR’s listeners—you made this new homepage possible! If you have questions, comments, complaints, or suggestions about the new homepage then please contact VPR directly with your feedback (although you’re welcome to comment here as well).

This bill proposes to require the department of information and innovation to advise against the acquisition by a state agency of proprietary software or software that operates by means of proprietary protocols when acceptable functionality can be achieved with open source software and open standard protocols and to ensure that the expected total life-cycle costs of software are fully evaluated prior to acquisition.

Overall I see this as a positive move for the State of Vermont. It does not seem to put the requirement for open source and open standards ahead of the particular business needs at hand since it only “advise[s] against the acquisition by a state agency of proprietary software or software that operates by means of proprietary protocols when acceptable functionality can be achieved with open source software and open standard protocols.” However, Mitch Lieberman points out:

@BradleyHolt functionality does not directly = biz value IMHO, biz value needs to include ROI of functions, thus includes TCO

I agree with Mitch that the bill could do a better job of addressing not just functionality, but also return on investment (ROI) and total cost of ownership (TCO). These should be the primary considerations of any responsible organization in deciding what software to use. However, since it is the public’s money being used to purchase this software it is worth considering whether our resources are being used to further entrench proprietary software and standards, or whether we are instead enriching the public commons by supporting open source software and open standards.

In addition to the above, there are some details of the bill that demonstrate some possible misunderstandings of how open source software works. While it’s not anywhere in its text, the bill is described as being “An act relating to comparing the costs and benefits of downloading free open source software with purchasing proprietary computer software.” There are a couple of misconceptions in this one statement. First, cost should not always be the primary motivation behind using open source software: the freedoms it gives you are just as important. In fact, not all proprietary software costs money and not all open source software is available free of charge. Second, “downloading” is simply a means of distributing software. Proprietary software can often be downloaded. Open source software is sometimes distributed on a physical medium.

It’s not stated explicitly in the bill, but people sometimes think that if they release their own software under an open source license they are somehow giving up control of the licensed software project. While there are many benefits of giving up control (which are outside the scope of this blog post), it’s worth noting that you don’t have to. When you license your software you still own the copyright on that software. You can still do whatever you want with that software, including release it under a different license later. Once you accept outside contributions then you need to honor whatever license has been given to you be your contributors (unless they agree to transfer copyright to you, but this can be tricky). It’s your choice whether or not to accept those contributions in the first place. You are still completely in control (as long as you want to be) and have simply given others the possibility of benefiting from your work.

Jason and I started Found Line almost five years ago and I’m looking forward to helping him and Liz continue to grow the business. We’ve got some really great clients and I’m very happy to have more time to focus on their projects. I’m also excited about the opportunity to work with free/open source software and open standards full time. So, time to go catch up on some work!

I was thinking about that question after the show when I remembered something: there’s a ton of free/open source code in the iPhone and iPod touch. These devices run a version of OS X, parts of which come from the free/open source FreeBSD operating system. Safari, the device’s web browser, uses the free/open source web browser engine WebKit. Contrary to popular belief, many free/open source software licenses allow you to make proprietary derivative works. Apple does contribute software back to the free/open source community even though they are not always legally required to.

My point is not to praise Apple’s contributions. Likely these contributions are beneficial to them. If they branch a free/open source project and their branch strays too far from the original, they will have a hard time incorporating new features from these projects. Linus Torvalds (credited with developing the Linux kernel) recently commented on Microsoft’s contribution of code to the Linux kernel:

I agree that it’s driven by selfish reasons, but that’s how all open source code gets written! We all “scratch our own itches”. It’s why I started Linux, it’s why I started git, and it’s why I am still involved. It’s the reason for everybody to end up in open source, to some degree.

Apple and other proprietary software companies (maybe even Microsoft) are starting to demonstrate an interesting symbioses with free/open source software. While I do think that free/open source software is better for “consumers” and beneficial to society in general I don’t think any of that is a necessary argument in support of free/open source software. I think that it is an approach that will ultimately create more value and be better at sustaining innovation than proprietary software. Smart software companies are starting to recognize this. Apple has reduced their upfront and ongoing research and development costs by tapping into free/open source software, even if the products they deliver to customers are proprietary. They also seem to understand the value of releasing some of their own software and modifications under free/open source licenses: they will get back more than they put in and the reward is greater than the risk.

My administration worked closely with the Open Government Committee formed by resolution of the City Council. In response to committee recommendations, we’ve included more information about public meetings on the City web-site and the Planning Department added a new section about proposed and pending zoning amendments. During the campaign it was clear that people want to know more about the activities of each city department. Over the next year we’ll look closely at how to finance and build a more uniform, responsive and user-friendly City website that will keep you better informed about public meetings and how you can participate in the process of government.

I’m all for open government and the City of Burlington, Vermont could certainly use a better website. I’d like to give the city some advice on its priorities in building a better website. This blog post is my first round of advice, but I’d like to hear reactions from other web people in and around Burlington — or anyone else who has an interest in our city having a better website! This advice is based on the goals outlined by Mayor Bob Kiss: that the website is inline with the spirit of Open Government, is responsive, and is user-friendly.

First, the website must be accessible and use open standards. I put these two together because by using open standards the website can be accessible. Specifically, the website should use strictly semantic XHTML. This allows for its use in the widest range of user agents (browsers) by the widest range of user abilities (as opposed to disabilities). Its presentation should be progressively enriched through CSS and its behavior should be progressively enhanced through unobtrusive JavaScript. Progressive enhancement means that the website is useful, at its core, without any CSS or JavaScript. It becomes more useful through the use of CSS and then, perhaps, even more useful through the use of JavaScript. However, neither CSS nor JavaScript should be required to access the website’s content.

Second, the website should be built using free (as in freedom, not cost) and open source software. It is our government’s responsibility to use non-proprietary software wherever possible. When it comes to building websites and web applications, there are no excuses for not using free and open source software. The capabilities of the open web platforms available equal, or exceed, the capabilities of proprietary web platforms. Any custom software developed for the city’s website must be licensed to the city under a free and open source license (this does not mean that the city shouldn’t pay for this software to be developed).

Third, the city should embrace the concept of open linked data. This means that the city should publish data, not just content. This data should be both human and machine readable. This allows third parties to access raw data and repurpose this data in new and interesting ways. To this end, data and content should be either put in the public domain (which is probably the case already) or use a Creative Commons license. Where possible, the city’s website should consume open data published by others rather than duplicating effort. There are several technical approaches to publishing and consuming open linked data including Microformats and the Semantic Web.

I hope these recommendations are useful. Once I’ve heard some feedback from other web people, I’ll forward these recommendations to my city councilors and the mayor. Hopefully they’ll find it helpful in improving the city’s website!

There’s one catch. There are many web sites and web applications built on the incorrect assumption that the web is a visual medium. The web designers and web developers who build these web sites and web applications break the web. In order to not break the web, web designers and web developers need to use open standards starting with plain old semantic (X)HTML. This semantic (X)HTML can then be progressively enriched/enhanced using CSS (for presentation) and JavaScript (for behavior). This approach works well with the concept of One Web which “means making, as far as is reasonable, the same information and services available to users irrespective of the device they are using.”

Craig Cook has a good article about How to Grok Web Standards in which he addresses many of these ideas and more. If you still think I’m crazy, hopefully it’s not because you think I’m wrong about the web not being a visual medium. If you don’t think I’m crazy then maybe in another blog post I’ll talk about why I think that content management systems (CMSs) in general, and WYSIWYG editors specifically, break the web.

Google has an interesting project called the Social Graph API that lets developers “utilize public connections their users have already created in other web services. It makes information about public connections between people easily available and useful.” It uses public XFN as well as FOAF data to accomplish this. FOAF is, interestingly enough, an example of something designed around the W3C’s Semantic Web specification. Here’s a video that explains a bit about how Google’s Social Graph API works:

The power of this open data is another example of why facebook won’t last as a walled garden. I don’t expect existing social networking sites to simply disappear. However, open standards like XFN and FOAF will help level the playing field amongst social networks by allowing even small social networks to benefit from the network effect. Having lots of social networks trying out different things can only be good for innovation in the long run.