Mirko Kiefer Rotating Header Image

Spawning a Node

I just published the source of what we’ve extracted out of whats been at the core of LivelyCouch.
The result is a tiny generic library that gives you only a single function – “spawn”.
It allows you to spawn a Javascript function in a new Node.js instance and returns you an EventEmitter to communicate with it.

You can find more details on github:
https://github.com/livelycode/spawn.js

Hopefully later tonight I will find some time to explain what drove us to implement this library and how we got there by basically rewriting LivelyCouch two times :)

Getting rid of C for-loops in Objective-C

Ever been annoyed by having to use C for-loops to do something simple like filtering or collecting objects from NSArray? Blocks finally allow you to get rid of them entirely!
As a Smalltalk lover I really had to write this simple library:

    [someArray forEach:^(id each) {
      // do something with each element
    }];

    NSArray* filteredArray = [someArray filter:^(id each) {
      return [each isEqual: aCoolObject];
    }];

    NSArray* collectedArray = [someArray collect: ^(id each) {
      if([each isEqual: aCoolObject]) {
        return @"cool";
      } else {
        return @"not cool";
      }
    }];

    [someDictionary keysAndValues: ^(id key, id value) {
      //do something with the current key and value
    }];

Fork the source on github:
http://github.com/livelycode/objc-prettyIterators

LivelyCouch source is available

Johannes Auer and I made great progress on the LivelyCouch framework I’ve written about before.

LivelyCouch now allows you to:

  • store your entire Node.js web application in CouchDB, being able to replicate it
  • changes on your source files are automatically written to CouchDB using a filesystem watcher
  • use “Handlers” to write your own views in Node.js
  • keep server-side code decoupled in small “Workers” that communicate through Events via publish/subscribe

Thanks to Johannes’ help we added a pattern-matching mechanism to Event subscriptions, today. This gives you maximum flexibility when wiring Events and Workers together.

We are now working on some tutorials to get you started on the framework.
Adventurous developers can check out our source at Github already now:
https://github.com/livelycouch/LivelyCouch

Let us know if you have any problems – we hope to get lots of feedback!

Introducing LivelyCouch Part 2 – Events and Workers

The LivelyCouch Event System can be used in your apps whenever you want to trigger some code that should run in the background. This could for example be:

  • sending out e-mails
  • resizing images
  • watching a database for changes
  • triggering an action every minute

The only means of communication in the Event System are HTTP messages.
You start with defining LivelyEvents as documents in an events database that listen to certain HTTP messages. Each Event document in turn defines which pieces of code should be executed when they are triggered.
These pieces of code are managed in so called LivelyWorkers. Workers reside as documents in their own database – similar to Handlers they store their code in the form of attachments.

Lets look at a sample Event document that triggers a Worker to send out e-mails:

{
   "_id": "email_event",
   "trigger": {
       "/send_email": {
           "method": "GET"
       }
   },
   "workers": {
       "send_email": {
           "recipient": "support@yourcompany.com",
       }
   }
}

This event would be triggered by a request of the form:
http://127.0.0.1:5984/_events/email_event

You see that it passes a “recipient” argument to the “send_email” worker – when the worker has been started, these arguments combined with the HTTP message are sent to him using stdin.

The worker document in the worker database is quite simple:

{
   "_id": "send_email",
   "arguments": {
   },
   "delegate": "sendmail.js",
   “type”: “node”,
   "_attachments": {
       "sendmail.js": {...},
      “someOtherScript.js”: {...}
   }
}

The delegate property defines which of the attached scripts should be executed – other scripts may be attached because they are required dependencies.
The type property tells the event system that sendmail.js is a Node.js script – it would be easy to implement workers in any other language as long as necessary interpreters are installed on the system.
When you write your own workers you only need to make sure you listen on stdin for incoming data. The Event System manages the rest like starting the Worker when triggered through an Event, sending it data and stopping the Worker when its no longer needed.

When the LivelyCouch code is ready to release I will work on some more practical tutorials. But I hope this explains the main concepts and look forward to suggestions and critique.

Introducing LivelyCouch Part 1 – writing Node.js handlers

LivelyCouch is all about making life as a CouchDB developer easier. It is designed to fill the gap between a CouchApp and a full fledged web application.
Being all written in Node.js it allows you to stick to Javascript throughout your code. We are currently working on getting the source ready to release.

LivelyCouch consists out of two major parts – LivelyHandler and the Lively Event System. They both run as Node scripts behind CouchDB using the new externals and http proxy modules which is available in CouchDB trunk by now.
In this part I will focus on how handlers work – part 2 will explain the event system a bit more.

A LivelyHandler basically allows you to do all the things you can’t do in Design Documents. That could be for example:

  • Chaining multiple CouchDB views
  • Doing requests across Databases
  • Combining multiple outputs of show functions

LivelyCouch makes it easy to write such handlers – here is a simple example:

exports.run = function(parameters, response, handlerLib) {
  var client = handlerLib.couchdb.createClient(5984, '127.0.0.1', 'user', 'password');
  var userDb = client.db('people');
  var hairColour = parameters.query.haircolour;
  var startAge = parameters.query.startage;
  var endAge = parameters.query.endage;

  // calling a view to get all people between 30 and 40
  userDb.view('my-design-doc', 'people-per-age', {startkey=startAge, endkey=endAge}, function(data) {
    // filtering all rows that have the specified hair colour:
    var filteredRows = data.rows.filter(function(row) {
      return row.value.haircolor == hairColour;
    });
    // writing out all filtered rows to the response:
    response.writeHead(200, {'Content-Type': 'text/plain'});
    response.write(JSON.stringify(filteredRows));
    response.end();
  })
}

All Handler follow this pattern – you simply export a “run” function in your Node script and LivelyCouch will call it passing it

  • the request parameters
  • a response you can write on
  • a handlerLib which contains useful modules like node-couchdb and mustache

To be able to execute it through a HTTP request, you simply upload the file as an attachment to a handler document in the “lively_handler” database.
The structure of such a handler document looks like the following:

{
  “id”: “your_app”,
  “_attachments”: [
    “filtered_people.js”: {...},
    “another_handler.js”: {...}
  ],
  “mapping”: {
    “/filtered_people”: “filtered_people.js”,
    “/another_path”: “another_handler.js”
  }
}

Through the mapping property you can define which handler binds to which URL path – all URLs are automatically prefixed with “_handler” and the name of the handler document.
So to call our filtered_people handler you would request the URL:
http://127.0.0.1:5984/_handler/your_app/filtered_people?haircolour=blond&startage=30&endage=40

Node.js and Amazon Web Services

I got pretty excited about Node.js during the last day, especially after discovering that there are already a whole bunch of modules available for it:

http://github.com/ry/node/wiki/modules

I needed clients for both Amazon EC2 and the Product Advertising API so I had a closer look at Blake Mizerany’s swirl-node.
As swirl-node only supports EC2 calls I decided to code my own little module.  I thought it would be best to have a generic part every Amazon API requires and more specific parts for EC2, Product Advertising and what else I may need in the future.

After a few hours of hacking and browsing the Node.js documentation I finally got it working.

Some simple usage examples:

var aws = require("./lib/node-aws");

ec2 = aws.createEC2Client(yourAccessKeyId, yourSecretAccessKey);
ec2.call("DescribeInstances", {}, function(result) {
console.log(JSON.stringify(result));
})

Returns you details about your EC2 instances:

[...]
{"item":{
"instanceId":"i-acb2d1db","imageId":"ami-03765c77",
"instanceState": {"code":"80","name":"stopped"},
"privateDnsName":{},"dnsName":{},
"reason":"User initiated (2010-07-28 19:37:54 GMT)"
[...]

or when using the Product Advertising API:

prodAdv = aws.createProdAdvClient(yourAccessKeyId, yourSecretAccessKey, yourAssociateTag);
prodAdv.call("ItemSearch", {SearchIndex: "Books", Keywords: "Javascript"}, function(result) {
console.log(JSON.stringify(result));
})

In fact returns you a long list of Books with details…

Everyone who needs something similar can give it a try at github and hopefully give me feedback:
http://github.com/livelycode/aws-lib

Everything is an Object in Javascript

Most developers have heard about “object orientation” and many believe it has something to do with what are called “classes”, “instances” and “class/instance methods” in Java or C++. In fact true object orientation has very little to do with these features – at least not in the sense the original “inventors” have seen this concept. But more on this in a later post.

For now I just want to share what I have learned about Javascript’s approach to OOP today. In Javascript essentially everything is an Object.

Javascript doesn’t have “classes”, it is a prototype-based language which means objects are created by cloning other objects:

var myNewObject = new Object();

Now, I can do whatever I want with this cloned object – for example I can give it new properties:

myNewObject.colour = ‘blue’;
myNewObject.anotherFancyAttribute = ‘blablub’;

You can also create functions:

function doSomething() { … };

This looks like functions are something different to objects – just like methods in Java are different to objects. But actually “functions” are just objects, too. In fact the function declaration above is just a shortcut for:

var doSomething = function() {…};

I can even add this function-object as a property to my object above:

myNewObject.doSomething = doSomething;

…and get essentially whats called a “method” in Java – I can invoke this function on the object and it will evaluate within the myNewObject context.

Realizing this everything-is-an-object concept is of great value if you want to do more advanced programming in Javascript. For me being in love with Smalltalk and having learned to appreciate the simplicity of a pure object language it is quite motivating to realize that Javascript isn’t as ugly as many tell you…

Learn Javascript!

There is hardly any other programming language that is as little understood as Javascript but still as widely used. Douglas Crockford made a good point on saying that everybody believes he can use Javascript without actually learning it. Until today there are very few applications written entirely in Javascript. Google tries to change that with Google Web Toolkit which allows writing entire webapps in Java and compiling them to pure Javascript.
For many developers this has the benefit of staying with a language and development environment they are used to.

Why is everyone trying to get around writing code in Javascript itself? Until recently I belonged to the same crowd, avoiding to write Javascript as long as possible and if unavoidable just copying and pasting some promising scripts from the web. Somehow Javascript got the image of being a “toy language” that “serious developers” don’t rely on.

But think about it for a second. If there is one language that we will have to rely on for the foreseeable future it is going to be Javascript. Nobody would seriously expect to see a stop of the internet’s growth anytime soon. Nobody can seriously hope that whats called a “webbrowser” will disappear as the gatekeeper to the web on everyones computer. And as Javascript is the only language that runs in a webbrowser, knowing this language in detail is a good investment in a developer’s future.

I will myself get started with Javascript in the next weeks and regularly post enlightenments here.

Google Wave – why replace e-mail?

I finally got a Google Wave Developer account and will have a closer look at the project during the next weeks.

The idea behind Google Wave is to reinvent e-mail the way it would look like when invented today. Instead of using plain and static text/html messages, people communicate in a much more dynamic way on the web today. When communicating with friends e-mail is hardly used anymore. Services like Facebook or Twitter allow a much more lightweight way of sending messages and make it easier to interact with multiple people. In your e-mail account you have no real way to filter incoming messages on their context. No matter if you are interested in an e-mail you are going to get it into your postbox. In e-mail you can not define the social environment you want to communicate in. You can not define a group of people you want to get started on a discussion. Of course you can set up mailing lists, of course you can CC mails to many people – but its tedious. You do not have a simple way to follow a certain conversation. In Facebook you can choose your Friends, in Twitter you choose whom to follow. Gmail is already going into the direction of leveraging e-mail by grouping messages to threads – but even Gmail can only guess which mails belong to what conversation by looking at the sender and subject. What if someone wants to add comments to earlier discussions? He would have to first look up that old mail, copy the relevant part of the message and paste it into a new mail – the whole context is lost. The receiver of that mail would probably have to look up the old mails, as well, to find relevant parts of the earlier discussion.
Most social networks sites also allow a seamless switch between simple message sending and live chat. This is not possible in e-mail either – you would have to run a chat tool in addition. This works but what if you want to continue a discussion from an e-mail in live chat? You can not track this discussion in a continuous way – you have to check both, your old mails and your chat log.
Finally there is Google Docs which is a great way of collaboratively working on documents – even in real-time and with the option to track all changes anyone made. This is an incredible improvement on the traditional way of sending around documents, let everyone make their changes, send them back, integrate all those changes, send it out again….

In my eyes these are the main drivers behind Google Wave – ease of communication, interacting with multiple people, switching between message sending and live communication, collaboratively editing and reviewing changes.

Solitude

“Others inspire us, information feeds us, practive improves our performance, but we need quiet time to figure things out, to emerge with new discoveries, to unearth original answers” Ester Buchholz