Bradley Holt » CouchDB

The Future of CouchDB and Couchbase Server

Bradley Holt — Thu, 05 Jan 2012 19:42:03 +0000

“What’s the future of CouchDB? It’s Couchbase.” —Damien Katz

“The future of CouchDB is CouchDB.” —Noah Slater

First of all, don’t panic. The Apache CouchDB project is thriving. There are plenty of core contributors who are dedicated to the project. If you’re using CouchDB, you’re fine. If you’re considering using CouchDB, you’ll be fine. CouchDB has a bright future ahead of it. Couchbase has repeatedly said that they’re not the CouchDB company, so this shouldn’t be a surprise to anyone.

My interpretation of this announcement is that CouchDB is forking. Going forward, you’ll have two choices, either Apache CouchDB or Couchbase Server. The road map for Apache CouchDB will continue to be determined by community consensus. The road map for Couchbase Server will be determined by Couchbase, the company.

CouchDB has two futures: CouchDB and Couchbase Server. Traditionally, forking a project in this manner has been considered a Bad Thing. From The Jargon File entry for forked:

An open-source software project is said to have forked or be forked when the project group fissions into two or more parts pursuing separate lines of development (or, less commonly, when a third party unconnected to the project group begins its own line of development). Forking is considered a Bad Thing — not merely because it implies a lot of wasted effort in the future, but because forks tend to be accompanied by a great deal of strife and acrimony between the successor groups over issues of legitimacy, succession, and design direction. There is serious social pressure against forking. As a result, major forks (such as the Gnu-Emacs/XEmacs split, the fissionings of the 386BSD group into three daughter projects, and the short-lived GCC/EGCS split) are rare enough that they are remembered individually in hacker folklore.

In my humble opinion, forking should no longer be considered a Bad Thing. Distributed revision control systems, such as Git, have made it much easier to merge changes back-and-forth between forked projects. If an individual or organization wants to take a project in a different direction, great! It sounds like Couchbase Server will be designed to meet a set of needs that are not being met by CouchDB. If so, a fork is perfectly appropriate as now both sets of needs can be met.

CouchApps at CouchConf NYC

Bradley Holt — Thu, 27 Oct 2011 20:05:52 +0000

Here are the slides from Monday’s CouchConf NYC session on CouchApps with CouchDB, JavaScript and HTML5:

CouchConf NYC CouchApps

View more presentations from Bradley Holt

Learning CouchDB at ZendCon 2011

Bradley Holt — Mon, 17 Oct 2011 20:10:03 +0000

Here are the slides from today’s ZendCon tutorial on Learning CouchDB:

ZendCon 2011 Learning CouchDB

View more presentations from Bradley Holt

You can instead download the PDF version, if you’d prefer.

If you were in this session, please give me feedback on Joind.in.

CouchApps at jQuery Conference Boston 2011

Bradley Holt — Sun, 02 Oct 2011 17:15:51 +0000

Here are the slides from today’s jQuery Conference presentation on CouchApps with CouchDB & jQuery:

jQuery Conference Boston 2011 CouchApps

View more presentations from Bradley Holt

If you were in this talk, please give me feedback on SpeakerRate.

Speaking at CouchConf NYC

Bradley Holt — Thu, 29 Sep 2011 16:33:58 +0000

I’m pleased to announce that I’ll be speaking at CouchConf New York City on October 24, 2011. This event is part of the CouchConf World Tour presented by Couchbase. My talk will be on CouchApps with CouchDB, JavaScript and HTML5. From the talk description:

In this talk we’ll see how to build CouchApps using CouchDB, Javascript, and HTML5. We’ll look at related tools such as the couchapp command ine tool, the Evently jQuery plugin, the CouchDB API jQuery plugin, the CouchApp Loader, Pathbinder, and the Mustache templating framework.

CouchDB and Domain-Driven Design

Bradley Holt — Tue, 02 Aug 2011 15:30:27 +0000

I’ve found CouchDB to be a great fit for domain-driven design (DDD). Specifically, CouchDB fits very well with the building block patterns and practices found within DDD. Two of these building blocks include Entities and Value Objects. Entities are objects defined by a thread of continuity and identity. A Value Object “is an object that describes some characteristic or attribute but carries no concept of identity.” Value objects should be treated as immutable.

Aggregates are groupings of associated Entities and Value Objects. Within an Aggregate, one member is designated as the Aggregate Root. External references are limited to only the Aggregate Root. Aggregates should follow transaction, distribution, and concurrency boundaries. Guess what else is defined by transaction, distribution, and concurrency boundaries? That’s right, JSON documents in CouchDB.

Let’s take a look at an example Aggregate, that representing a blog entry and related metadata. Note that the following UML diagrams are for classes in PHP, but it should be easy enough to translate these examples to any object-oriented programming language. We’ll start with the Entry Entity, which will serve as our Aggregate Root:

-----------------------------------------
|                 Entry                 |
-----------------------------------------
|+ id : string                          |
|+ rev : string                         |
|+ title : Text                         |
|+ updated : Date                       |
|+ authors : Person[*]                  |
|+ content : Text                       |
-----------------------------------------
|+ __construct(entry : array) : void    |
|+ toArray() : array                    |
-----------------------------------------

The Text Value Object:

----------------------------------------------
|                    Text                    |
----------------------------------------------
|- type : string                             |
|- text : string                             |
----------------------------------------------
|+ __construct(type : string, text : string) |
|+ toArray() : array                         |
----------------------------------------------

The Date Value Object:

--------------------------------------
|                Date                |
--------------------------------------
|- timestamp : integer               |
--------------------------------------
|+ __construct(timestamp : integer)  |
|+ __toString() : string             |
--------------------------------------

The Person Value Object:

-------------------------------------------------------------
|                           Person                          |
-------------------------------------------------------------
|- name : string                                            |
|- uri : string                                             |
|- email : string                                           |
-------------------------------------------------------------
|+ __construct(name : string, uri : string, email : string) |
|+ toArray() : array                                        |
-------------------------------------------------------------

I recommend serializing each Aggregate, starting with the Aggregate Root, into a JSON document. Control access to Aggregate Roots through a Repository. The toArray() methods above return an associative array representation of each object. The Repository can then transform the array into JSON for storage in CouchDB. Let’s take a look at the EntryRepository:

---------------------------------
|        EntryRepository        |
---------------------------------
|                               |
---------------------------------
|+ get(id : string) : Entry     |
|+ post(entry : Entry) : void   |
|+ put(entry : Entry) : void    |
|+ delete(entry : Entry) : void |
---------------------------------

Here’s an example of what the Aggregate’s object graph might look like, serialized as a JSON document:

{
    "_id": "http://bradley-holt.com/?p=1251",
    "title": {
        "type": "text",
        "text": "CouchDB and Domain-Driven Design"
    },
    "updated": "2011-08-02T15:30:00+00:00",
    "authors": [
        {
             "name": "Bradley Holt",
             "uri": "http://bradley-holt.com/",
             "email": "bradley.holt@foundline.com"
        }
    ],
    "content": {
        "type": "html",
        "text": "I've found CouchDB to be a great fit for…"
    }
}

You can also provide access to CouchDB views through Repositories. In the above example, this could be through the addition of an index(skip : integer, limit : integer) : Entry[*] method to the the EntryRepository (note that this is a naive pagination implementation, especially on large data sets—but that’s beyond the scope of this blog post). For more complex views, you may want to create a separate Repository for each CouchDB view.

Addressing the NoSQL Criticism

Bradley Holt — Sat, 30 Jul 2011 16:59:10 +0000

There were quite a few NoSQL critics at OSCON this year. I imagine this was true of past years as well, but I don’t know that first hand. I think there are several reasons behind the general disdain for NoSQL databases.

First, NoSQL is horrible name. It implies that there’s something wrong with SQL and it needs to be replaced with a newer and better technology. If you have structured data that needs to be queried, you should probably use a database that enforces a schema and implements Structured Query Language. I’ve heard people start redefining NoSQL as “not only SQL”. This is a much better definition and doesn’t antagonize those who use existing SQL databases. An SQL database isn’t always the right tool for the job and NoSQL databases give us some other options.

Second, there are way too many different types of databases that are categorized as NoSQL. There are document-oriented databases, key/value stores, graph databases, column-oriented databases, in-memory databases, and other database types. There are also databases that combine two or more of these properties. It’s easy to criticize something that is vague and loosely defined. As the NoSQL space matures, we’ll start to get some more specific definitions, which will be much more helpful.

Third, at least one very popular vendor in the NoSQL space has a history of making irresponsible claims about their database’s capabilities. Antony Falco of Basho (makers of Riak) has a great blog post on the topic: See It’s Time to Drop the “F” Bomb – or “Lies, Damn Lies, and NoSQL.” If you care about your data, please read Tony’s blog post. It’s unfortunate that the specious claims of a few end up making everyone in the NoSQL space look bad.

I also want to address some of the specific criticisms that I’ve heard of NoSQL, as they apply (or don’t apply) to CouchDB (I’m not familiar enough with other NoSQL databases to talk about those).

SQL Databases Are More Mature

This is absolutely true. If you pick a NoSQL database, you should do your homework and make sure that your database of choice truly respects the fact that writing a reliable database is a very difficult task. Most of the NoSQL databases take the problem very seriously, and try to learn from those that have come before them. But why create a new type of database in the first place? Because an SQL database is not the right solution to every problem. When all you have is a schema, everything looks like a join. The data model in CouchDB (JSON documents) is a great fit for many web applications.

SQL Scales Just Fine

This is also true. If you’re picking a NoSQL database because it “scales”, you’re likely doing it wrong. Scaling is typically more aspiration than reality. There are many other factors to consider and questions to ask when choosing a database technology other than, “does it scale?” If you do actually have to scale, then your database isn’t going to magically do it for you. You can’t abstract scaling problems to your database layer. However, I will say that many NoSQL databases have properties (such as eventually consistency) that will make scaling easier and more intuitive. For example, it’s dead simple to replicate data between CouchDB databases.

Atomicity, Consistency, Isolation, and Durability (ACID)

CouchDB is ACID compliant. Within a CouchDB server, for a single document update, CouchDB has the properties of atomicity, consistency, isolation, and durability (ACID). No, you can’t have transactions across document boundaries. No, you can’t have transactions across multiple servers (although BigCouch does have quorum reads and writes). Not all NoSQL databases are durable (at least with default settings).

If you want the best possible guarantee of durability, you can change CouchDB’s delayed_commits configuration option from true (the default) to false. Basically, this will cause CouchDB to do an explicit fsync after each operation (which is very expensive and slow). Note that operating systems, virtual machines, and hard drives often lie about fsync, so you really need to research more about how your particular system works if you’re concerned about durability. If you think your write speeds are too good to be true, they probably are.

If you leave delayed commits on, CouchDB has the option of setting a batch=ok parameter when creating or updating a document. This will queue up batches of documents in memory and write them to disk when a predetermined threshold has been reached (or when triggered by the user). In this case, CouchDB will respond with an HTTP response code of 202 Accepted, rather than the normal 201 Created, so that the client is informed about the reduced integrity guarantee.

Consistency Checks

At least one NoSQL database requires a consistency check after a crash (guess which one). This can be a very slow process, causing additional downtime. CouchDB’s crash-only design and append-only files means that there is no need for consistency checks. There’s no shut down process in CouchDB—shutting it down is the same as killing the process.

Compaction

CouchDB’s append-only files do come at a cost. That cost is disk space and the need for compaction. If you don’t compact your database, it will eventually fill up your hard drive. There is no automatic compaction in CouchDB. Compaction is triggered manually (it can easily be automated through a cron job) and should be done when the database’s write load is not at full capacity.

MapReduce is Limiting and Hard to Understand

It can take some time to get up to speed with MapReduce views in CouchDB. However, it’s not a very difficult concept to understand and most developers are already proficient with JavaScript (the default language for Map and Reduce functions in CouchDB). There’s a lot you can do with MapReduce, but there are some limitations. Views are one dimensional so full text indexing and geospatial data are difficult (if not impossible) to index. However, there are plugins for integrating with Lucene and ElasticSearch. For geospatial data, you can use GeoCouch.

No Ad Hoc Queries

This is a feature, not a bug. CouchDB only lets you query against indexes. This means that queries in CouchDB will be extremely fast, even on huge data sets. Most web applications have predefined usage patterns and don’t need ad hoc queries. If you need ad hoc queries, say for business intelligence reporting, you can replicate your data (using CouchDB’s changes feed) to an SQL database.

Building Indexes is Slow

If you have a large number of documents in CouchDB, the first build of an index will be very slow. However, each query after that will be very fast. CouchDB’s MapReduce is incremental, meaning new or updated documents can be processed without needing to rebuild the entire index. In most scenarios, this means that there will be a small performance hit to process documents that are new or updated since the last time the view was queried. You can optionally include the stale=ok parameter with your query. This will instruct CouchDB to not bother processing new or updated documents and just give you a stale result set (which will be faster than processing new or updated documents). As of CouchDB 1.1, you can include a stale=update_after parameter with your query. This will return a stale result set, but will trigger an update of the index (if necessary) after your query results are returned, bringing the index up-to-date for future queries by you or other clients.

No Schema

Some say that not having a schema is a problem. Sure—if you have structured data, you probably want to enforce a schema. However, not all applications have highly structured data. Many web applications work with unstructured data. If you’ve encountered any of the following, you may want to consider a schema-free database:

You’ve found yourself denormalizing your database to optimize read performance.
You have rows with lots of NULL values because many columns only apply to a subset of your rows.
You find yourself using SQL antipatterns such as entity-attribute-value (EAV), but can’t find any good alternatives that fit with both your domain and SQL.
You’re experiencing problems related to the object-relational impedance mismatch. This is typically associated with use of an object-relational mapper (ORM), but can happen when using other data access patterns as well.

I’ll add that you can enforce schemas in CouchDB through the use of document update validation functions.

Anything Else?

Did I miss anything? What other criticisms exist of NoSQL databases? Please comment and I’ll do my best to address each.

CouchApps at OSCON 2011

Bradley Holt — Wed, 27 Jul 2011 23:30:23 +0000

Here are the slides from today’s OSCON presentation on CouchApps with CouchDB, JavaScript & HTML5:

Bradley Holt » CouchDB

The Future of CouchDB and Couchbase Server

Top Five Posts of 2011

5. CouchDB and Domain-Driven Design

4. Addressing the NoSQL Criticism

3. Testing PHP 5.4

2. Exploring RabbitMQ and PHP

1. CouchDB jQuery Plugin Reference

CouchApps at CouchConf NYC

Learning CouchDB at ZendCon 2011

CouchApps at jQuery Conference Boston 2011

Speaking at CouchConf NYC

CouchDB and Domain-Driven Design

Addressing the NoSQL Criticism

SQL Databases Are More Mature

SQL Scales Just Fine

Atomicity, Consistency, Isolation, and Durability (ACID)

Consistency Checks

Compaction

MapReduce is Limiting and Hard to Understand

No Ad Hoc Queries

Building Indexes is Slow

No Schema

Anything Else?

CouchApps at OSCON 2011

Learning CouchDB at OSCON 2011