Following a suggestion from Kloanor on a Hacker News article I got a WebFaction account to play around with node.js on.

tl;dr It’s not too hard to set up.

Getting Ready

All of this software will be installed in my home directory, so there are a few things we need to do. First, I created a sources directory so that I would have my hands on exact copies of the installed software in the future, as well as a clean place to unpack and build them.

[littlef@web140 ~]$ mkdir sources
[littlef@web140 ~]$

Second, we need to create two custom applications for MongoDB and node.js using the WebFaction control panel.

This is pretty easy, just go to “Domains/Websites > Applications > Add New“. Now give it a name (I used mongodb_master) and select “Custom app (listening on port)” as the App Type.

Hit create and write down the port number it provides. Do this again for node.js. You can also take a moment to map these applications to a website and domain.

MongoDB

MongoDB is actually supported to some extent, with install instructions provided in the WebFaction Doc’s. I did this slightly different, so I’ll detail my version here.

Acquiring MongoDB

You can get MongoDB at http://www.mongodb.org/display/DOCS/Downloads. You’ll need the 32-Bit Linux version. At the time of this writing the most current stable release was 1.4.3.

[littlef@web140 ~]$ cd sources
[littlef@web140 sources]$ wget  http://downloads.mongodb.org/linux/mongodb-linux-i686-1.4.3.tgz
--2010-06-18 18:43:29--  http://downloads.mongodb.org/linux/mongodb-linux-i686-1.4.3.tgz
Resolving downloads.mongodb.org... 72.21.202.134
Connecting to downloads.mongodb.org|72.21.202.134|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 23709740 (23M) [application/x-tar]
Saving to: “mongodb-linux-i686-1.4.3.tgz”
 
100%[==============>] 23,709,740   202K/s   in 1m 55s
 
2010-06-18 18:45:29 (201 KB/s) - “mongodb-linux-i686-1.4.3.tgz” saved [23709740/23709740]
 
[littlef@web140 sources]$

Installing MongoDB

MongoDB comes pre-compiled, so installing it is as easy as unpacking and moving some directories.

[littlef@web140 sources]$ tar -zxf mongodb-linux-i686-1.4.3.tgz
[littlef@web140 sources]$ cd mongodb-linux-i686-1.4.3/
[littlef@web140 mongodb-linux-i686-1.4.3]$ ls
bin  GNU-AGPL-3.0  include  lib  README  THIRD-PARTY-NOTICES
[littlef@web140 mongodb-linux-i686-1.4.3]$ mv bin ~/
[littlef@web140 mongodb-linux-i686-1.4.3]$ mv lib ~/
[littlef@web140 mongodb-linux-i686-1.4.3]$ mv include ~/

Configuring MongoDB

In this case, MongoDB is configured at run time. But it will need a data directory. I chose to put this at ~/var/mongo/master/. You don’t have to do anything special, just make sure the directory exists, and is empty.

That’s it! MongoDB is installed. We’ll come back and fire it up after we get node.js installed.

Installing node.js

node.js is almost as easy as MongoDB. And while not documented anywhere, it does run just fine. The major point of contact here is a forum topic that details how to get it running. Again, I’m going to deviate a bit, but I’ll end up in roughly the same place.

Acquiring node.js

You can get node.js at http://nodejs.org/#download. It’s distributed as source, so there is nothing to pick and choose from. At the time of this writing the most current version was 0.1.98.

[littlef@web140 ~]$ cd sources
[littlef@web140 sources]$ wget http://nodejs.org/dist/node-v0.1.98.tar.gz
--2010-06-18 18:57:14--  http://nodejs.org/dist/node-v0.1.98.tar.gz
Resolving nodejs.org... 97.107.132.72
Connecting to nodejs.org|97.107.132.72|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3770749 (3.6M) [application/octet-stream]
Saving to: “node-v0.1.98.tar.gz”
 
100%[===========>] 3,770,749    151K/s   in 25s
 
2010-06-18 18:57:39 (147 KB/s) - “node-v0.1.98.tar.gz” saved [3770749/3770749]
 
[littlef@web140 sources]$

Easier done than said.

Installing node.js

You have to compile node.js, but this went perfectly clean for me. Just make sure you set the prefix when you configure, or you’ll have to do it all again.

[littlef@web140 sources]$ tar -zxf node-v0.1.98.tar.gz
[littlef@web140 sources]$ cd node-v0.1.98/
[littlef@web140 node-v0.1.98]$ ./configure --prefix=$HOME
Checking for program g++ or c++          : /usr/bin/g++                                                            
...snip...
Checking for fdatasync(2) with c++       : yes
'configure' finished successfully (5.495s)
[littlef@web140 node-v0.1.98]$ make
Waf: Entering directory `/home/littlef/sources/node-v0.1.98/build'                                                         
DEST_OS: linux                                                                                                     
...snip...
Waf: Leaving directory `/home/littlef/sources/node-v0.1.98/build'
'build' finished successfully (5m33.731s)
[littlef@web140 node-v0.1.98]$ make install

Configuring node.js

Nothing to configure! It should be ready, like, right now.

Running Your New Software

Okay, everything is installed. Let’s get stuff running.

Running MongoDB

Running MongoDB should only take two options. --dbpath and --port.

Go ahead and fire it up, then shut it down with ctrl-c.

[littlef@web140 ~]$ mongod --dbpath ~/var/mongo/master/ --port 39381
Fri Jun 18 19:11:24 Mongo DB : starting : pid = 3293 port = 39361 dbpath = ~/var/mongo/master master = 0 slave = 0  32-bit
 
** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
**       see http://blog.mongodb.org/post/137788967/32-bit-limitations for more
 
Fri Jun 18 11:11:24 db version v1.2.2, pdfile version 4.5
Fri Jun 18 11:11:24 git version: nogitversion
Fri Jun 18 11:11:24 sys info: Linux biber 2.6.26-2-amd64 #1 SMP Thu Feb 11 00:59:32 UTC 2010 i686 BOOST_LIB_VERSION=1_40
Fri Jun 18 11:11:24 waiting for connections on port 39361
^CFri Jun 18 11:11:27 got kill or ctrl c signal 2 (Interrupt), will terminate after current cmd ends
Fri Jun 18 11:11:27 now exiting
Fri Jun 18 11:11:27  dbexit:
Fri Jun 18 11:11:27      shutdown: going to flush oplog...
Fri Jun 18 11:11:27      shutdown: going to close sockets...
Fri Jun 18 11:11:27      shutdown: waiting for fs...
Fri Jun 18 11:11:27      shutdown: closing all files...
Fri Jun 18 11:11:27      closeAllFiles() finished
Fri Jun 18 11:11:27      shutdown: removing fs lock...
Fri Jun 18 11:11:27  dbexit: really exiting now
[littlef@web140 ~]

That’s great, but we can’t just stay logged in to SSH all the time. Let’s nohup it, background it and forget about it.

[littlef@web140 ~]$ nohup mongod --dbpath ~/var/mongo/master/ --port 39381 &
[1] 3299
nohup: ignoring input and appending output to `nohup.out
[littlef@web140 ~]$

Now we can query it by setting the port on mongo

[littlef@web140 ~]$ mongo --port 39381 noderegator
MongoDB shell version: 1.2.2
url: noderegator
connecting to: 127.0.0.1:39381/noderegator
type "exit" to exit
type "help" for help
> show collections
> exit
bye
[littlef@web140 ~]$

Running node.js

Before we can actually run node.js, we need to write a little application. In my case this will go in ~/webapps/noderegator_nodejs/app.js. Place yours wherever you set up your custom application for node.

Be sure to change the port number to the one issued to your application by WebFaction.

1
2
3
4
5
6
7
8
9
10
11

var http = require( 'http' ),
     sys = require( 'sys' );
 
http.createServer(
  function( request, response ) {
    sys.puts( 'Request!' );
    response.writeHead( 200, { 'Content-Type': 'text/plain' } );
    response.end( 'Hello World\n');
  }
).listen( 35408 );
sys.puts( 'Server running on port 35408' );

Now let’s fire that up and see it in the browser. ctrl-c to quit.

[littlef@web140 ~]$ cd webapps/noderegator_nodejs/
[littlef@web140 noderegator_nodejs]$ node app.js
Server running on port 35408
Request!
Request!
^C
[littlef@web140 noderegator_nodejs]$

If you got an nginx 503 error when you tried to visit the site, check that your port numbers are set up correctly.

All Done

That’s it, that’s all you need to know. Enjoy MongoDB and node.js on WebFaction. I hope it all goes well for you.

I never, ever remember the commands for ChanServ on Freenode. I’m always looking them up, so this time I made a quick cheat-sheet to help me out. You can basically follow this from top to bottom.

Register Your Channel

/msg ChanServ register #<channel> <password>

Tells ChanServ about your channel and makes you the founder.

Make ChanServ Stick Around

/msg ChanServ set #<channel> guard on

Makes ChanServ hang out in your channel.

Protect Your Topic

/msg ChanServ set #<channel> topiclock on

Makes it so you can only change the topic through ChanServ.

Set A Topic

/msg ChanServ set #<channel> topic <topic>

Set the channel topic via ChanServ.

Make Your Topic Sticky

/msg ChanServ set #<channel> keeptopic on

ChanServ keeps your topic even when there is no one in the channel.

Set The Channel URL

/msg ChanServ set #<channel> url <url>

Sets the URL for the channel if someone does an INFO query.

Set The Channel Description

/msg chanserv set #<channel> desc <description>

Sets the description for the channel if someone does an INFO or LIST query.

Sources:

• soundsnwaves.net
• dal.net

Prism is a sandboxed web-app tool, essentially it is Firefox for one application at a time. It’s been a while since I played with it, so I’m taking another look.

This should work on any Debian based system really, but I did it on Sidux. Here’s how I installed the latest version on my machine.

# wget http://prism.mozilla.com/downloads/1.0b1/prism-1.0b1.en-US.linux-i686.tar.bz2
--2010-01-26 16:01:08--  http://prism.mozilla.com/downloads/1.0b1/prism-1.0b1.en-US.linux-i686.tar.bz2
Resolving prism.mozilla.com... 63.245.208.216
Connecting to prism.mozilla.com|63.245.208.216|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9101612 (8.7M) [application/x-bzip2]
Saving to: “prism-1.0b1.en-US.linux-i686.tar.bz2”
 
100%[==========================================================================================================================================================================================================>] 9,101,612    212K/s   in 42s
 
2010-01-26 16:01:55 (213 KB/s) - “prism-1.0b1.en-US.linux-i686.tar.bz2” saved [9101612/9101612]
 
# mv prism /opt/prism-1.0b1
# ln -s /opt/prism-1.0b1/prism /usr/bin

At that point it should be in your path and ready to invoke!

I recently got interested in the lifestream application Storytlr.

One of the first things I wanted to do was add a plugin for github, as that is a fairly large part of my online life. Unfortunately there is very little documentation, which is understandable for a private project that just went open.

As such I had to work through some code, but all in all it wasn’t too tough. The source is very readable, just not set up with any auto-documentation.

So, as a way of giving back, I’m going to walk through the Storytlr plugin system, as I understand it. If you want to play along at home, you can grab the source from my github, and the commit we will be working from is 23136196

First things first, we need to copy off another plugin, that’s the easy way to get started. I cloned the RSS one, but they are all pretty similar to start with. Here’s what my final file tree looks like, it’s pretty similar to the starting tree, just with “Github” instead of “RSS”. This would reside at path_to_app/protected/application/plugins/github

.
├── database.sql
├── github.png
├── models
│   ├── GithubItem.php
│   └── GithubModel.php
└── views
    └── scripts
        ├── rss.phtml
        ├── story.phtml
        └── timeline.phtml

So what are each of these and what do we have to do to them? Well, I like to drive my development by my data structures, so let’s start with database.sql. This is pretty straightforward SQL, and I’ll highlight some key fields after you have a look.

database.sql

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

DROP TABLE IF EXISTS `github_data`;
SET @saved_cs_client = @@character_set_client;
SET character_set_client = utf8;
CREATE TABLE `github_data` (
  `id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
  `source_id` int(10) UNSIGNED NOT NULL,
  `github_id` varchar(255) NOT NULL,
  `title` text NOT NULL,
  `content` text,
  `repository` text,
  `link` varchar(255) NOT NULL,
  `published` varchar(45) NOT NULL,
  PRIMARY KEY USING BTREE (`id`),
  UNIQUE KEY `DUPLICATES` USING BTREE (`source_id`, `github_id`),
  FULLTEXT KEY `SEARCH` (`content`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
SET character_set_client = @saved_cs_client;

So what are our key items to look at here? Well, how about table naming convention? The existing tables are named as their plugin name, all lowercase, then _data. Why get fancy? Let’s use the convention.

What else is important? I kept the id and source_id pieces intact. These are actually fairly important, as I learned when I tried to make source_id into a varchar. source_id is actually the key that is tied to the sources table. That seems obvious now, but trust me, it didn’t when I was writing this the first time.

Aside from that, there isn’t too much to worry about. Just make sure to add something that you can use to identify individual updates on, github_id in this case, and then add your data fields and set your indexes. The naming of the FULLTEXT "SEARCH" key is probably important, though I didn’t test it any other way. You’ll probably find that all of your plugin tables will look more or less alike.

Now what? Well, let’s go ahead and define the model for an item, since we are already thinking about the data types. Here is a partial listing of GithubItem.php, have a look.

19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

class GithubItem extends SourceItem {
 
  protected $_prefix   = 'github';
 
  protected $_preamble = 'Github activity: ';
 
  public function getContent() { return $this->_data['content']; }
 
  public function getTitle() {
    $title = str_replace(
      $this->_data['repository'],
      '<a href="http://github.com/' . $this->_data['repository'] . '">' . $this->_data['repository'] . '</a>',
      html_entity_decode( strip_tags( $this->_data['title'] ) )
    );
    return $title;
  }
 
  public function getLink() { return $this->_data['link']; }
 
  public function getType() { return SourceItem::LINK_TYPE; }
 
  public function getBackup() {
    $item = array();
    $item['SourceID'] = $this->_data['source_id'];
    $item['Title'] = $this->_data['title'];
    $item['Content'] = $this->_data['content'];
    $item['Repository'] = $this->_data['repository'];
    $item['Link'] = $this->_data['link'];
    $item['Published'] = $this->_data['published'];
    return $item;
  }
 
}

This is a very stripped down implementation. All of your data is available to the views through the inherited SourceItem::toArray method, so you really only need to override a few methods and put in any mangle logic, as I did in the getTitle method.

One other method I was sure to override was getType. The types are enumerated in /protected/application/admin/models/SourceItem.php, and are as follows:

21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

  const IMAGE_TYPE  = 'image';
 
  const AUDIO_TYPE  = 'audio';
 
  const VIDEO_TYPE  = 'video';
 
  const STATUS_TYPE   = 'status';
 
  const BLOG_TYPE   = 'blog';
 
  const LINK_TYPE   = 'link';
 
  const OTHER_TYPE  = 'other';
 
  const STORY_TYPE  = 'story';

I am not sure how they play into the rendering process at this point, but better safe than sorry, right?

Let’s move on to the views. These are all very similar, and are only rendered in different, well, views. I’ll go over timeline.phtml, but if you copy the others you should be able to piece it together in a jiffy.

1
2
3
4

<div class="github">
  <div class="title"><?= $this->item->getTitle(); ?></div>
  <div class="content"><?= $this->item->getContent(); ?></div>
</div>

Pretty brutal, huh? You just get your title (mangled by GithubItem::getTitle in this case) and get your content, and you are as good as done.

Finally, I’m going to address the engine that drives all of this, GithubModel.php. A lot of this is just editing boilerplate, so let’s start with that. Comments are added in-line, but aren’t in the git source.

19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

class GithubModel extends SourceModel {
  // What is the table named?
  protected $_name   = 'github_data';
  // What is the plugin directory named?
  protected $_prefix = 'github';
  // What fields are searchable? This is comma delimited, i.e. "content, title"
  protected $_search  = 'content';
  // What is a format string for update tweets?
  protected $_update_tweet = "Did %d things at github.com on my lifestream %s";
  // What is the service name shown on the backend/widgets
  public function getServiceName() {
    return "Github";
  }
  // ?
  public function isStoryElement() {
    return true;
  }
  // What is the URL for the widget links?
  public function getServiceURL() {
    return 'http://github.com/' . $this->getProperty('username');
  }
  // Brief description for the admin interface
  public function getServiceDescription() {
    return "Github is social coding.";
  }
  // What is the name on the account (for the admin interface mostly)
  public function getAccountName() {
    if ($name = $this->getProperty('username')) {
      return $name;
    }
    else {
      return false;
    }
  }
  // What is the title of the account (for the front end)
  public function getTitle() {
    return $this->getServiceName();
  }
  // The initial data import function
  public function importData() {
    $items = $this->updateData();
    $this->setImported( true );
    return $items;
  }

Okay, now we are ready to dig into the meaty part, the data update functions. This is split into two parts, updateData and processItems. Here is a (hopefully) self-explanatory updateData implementation, more or less cloned from existing plugins.

51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

  public function updateData() {
    $url  = 'http://github.com/' . $this->getProperty('username') . '.atom';
 
    $curl = curl_init();
    curl_setopt($curl, CURLOPT_URL, $url);
    curl_setopt($curl, CURLOPT_HEADER, false);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($curl, CURLOPT_USERAGENT,'Storytlr/1.0');
 
    $response = curl_exec($curl);
    $http_code = curl_getinfo($curl, CURLINFO_HTTP_CODE);
    curl_close ($curl);
 
    if ($http_code != 200) {
      throw new Stuffpress_Exception( "Github returned http status $http_code for url: $url", $http_code );
    }
 
    if (!($items = simplexml_load_string($response))) {
      throw new Stuffpress_Exception( "Github did not return any result", 0 );
    }
 
    if ( count( $items->entry ) == 0 ) { return; }
 
    $items = $this->processItems($items->entry);
    $this->markUpdated();
    return $items;
  }

Pretty easy, right? Just acquire the content (any way you wish, this one uses cURL), parse it out into an array and pass it on. If all goes well in processItems, then mark it as a successful update with SourceModel::markUpdated. Let’s see what processItems does.

80
81
82
83
84
85
86
87
88
89
90
91
92
93
94

  private function processItems($items) {
    $result = array();
    foreach ($items as $item) {
      $data = array();
      $data['title'] = $item->title;
      $data['repository'] = substr( $item->title, strrpos( $item->title, ' ' ) + 1 );
      $data['published'] = strtotime( $item->published );
      $data['content'] = $item->content;
      $data['link'] = $item->link['href'];
      $data['github_id'] = $item->id;
      $id = $this->addItem( $data, $data['published'], SourceItem::LINK_TYPE, array( $data['repository'] ), false, false, $data['title'] );
      if ($id) $result[] = $id;
    }
    return $result;
  }

Again, pretty simple. We take the passed in items, mangle them as needed and stuff them into a data array for storage. About the only unknown there is the addItem call, which is in the SourceModel class. Let’s take a look at the first part of that to understand our parameters.

/protected/application/admin/models/SourceModel.php

168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183

  public function addItem($data, $timestamp, $type, $tags=false, $location=false, $hidden=false, $title=false) {
    $data['source_id']   = $this->_source['id'];
    $columns         = array();
    $keys            = array();
    $timestamp       = ($timestamp>=0) ? $timestamp : 0;
 
    foreach($data as $k => $v) {
      unset($data[$k]);
      if (!$v) continue;
      $columns[] = "$k";
      $keys[] = ":$k";
      $data[":$k"] = "$v";
    }
 
    $sql = "INSERT IGNORE INTO {$this->_name} (".implode(',', $columns).") "
       . "VALUES(".implode(',', $keys).")";

The most important thing for us to note are the names of the arguments: $data, $timestamp, $type, $tags, $location, $hidden, $title. These are self-explanatory and help us understand why the existing plugins pass what they do. Some other pieces to note is the override of source_id on line 169, and how it builds the query from your $data arguments, on lines 174-183. Naming matters!

So, now we are back to GithubModel, and we just have a few more methods to go. What remains below are the form generators and processing for the admin interface. Github only needs one piece of information to work, the username, so that’s all we are asking for below.

GithubModel.php

97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125

  public function getConfigForm($populate=false) {
    $form = new Stuffpress_Form();
 
    // Add the username element
    $element = $form->createElement('text', 'username', array('label' => 'Username', 'decorators' => $form->elementDecorators));
    $element->setRequired(true);
    $form->addElement($element);
 
    // Populate
    if($populate) {
      $values  = $this->getProperties();
      $form->populate($values);
    }
 
    return $form;
  }
 
  public function processConfigForm($form) {
    $values = $form->getValues();
    $update  = false;
 
    if($values['username'] != $this->getProperty('username')) {
      $this->_properties->setProperty('username',   $values['username']);
      $update = true;
    }
 
    return $update;
  }
}

getConfigForm creates the form fields using the Stuffpress_Form class, which is described in /protected/library/Stuffpress/Form.php. Github only has the one element, so it is added and set to required, then we are done.

processConfigForm is similarly simple, we get the value of our field, then make sure it is valid. If it is, we save it into our model’s properties, which is a SourcesProperties class, which is in turn a Stuffpress_Db_Properties class. Essentially think of it as a persistent key/value store. Or don’t think about it at all, just use it.

At this point you should have a working plugin!

Debugging Storytlr can be tough sometimes, so make sure your config file has debug = 1, and keep an eye on /protected/logs/.

If you have any questions, comments or corrections, please let me know!

Need to get the exact time that you visited a page in Firefox? I couldn’t find an easy way to look this up in the History interface, or anywhere else for that matter. I did however know that Firefox stores this kind of thing in sqlite3 databases. Here’s how I got what I needed.

First you have to find the sqlite databases, I’m on Linux so that would be in my home directory. The database you want is places.sqlite. Crack that open in sqlite3. Your command will differ as this is based on your profile name, mine is “gmail” so I ended up with g69ap5lc.gmail.

$ sqlite3 ~/.mozilla/firefox/g69ap5lc.gmail/places.sqlite

Be aware you have to shut down the Firefox instance first, because it locks the file. Make sure your privacy settings won’t erase it all when you shut it down! I had to change mine to “Remember history” first.

Next you need to find and grab the timestamp. This can be a chore if you don’t have the full URL. I was looking for the one from spiffie.org below.

sqlite>.headers on
sqlite>select * from moz_places;
id|url|title|rev_host|visit_count|hidden|typed|favicon_id|frecency|last_visit_date
1|http://www.mozilla.com/en-US/firefox/central/|/en-US/firefox/central/|moc.allizom.www.|0|0|0||140|
...
1366|http://spiffie.org/kits/usb7/driver_linux.shtml|Linux USB7 Driver|gro.eiffips.|1|0|0||100|1261169238197827

The column we are interested in is last_visit_date which is 1261169238197827 in our case. You can also list all the recent visits from the moz_historyvisits table with the id column.

sqlite> select * from moz_historyvisits where place_id = '1366';
id|from_visit|place_id|visit_date|visit_type|session
200|199|1366|1261169238197827|6|42

Now we need to convert that timestamp into something we can read (unless you are a super UNIX geek and can read timestamps). This instance is too precise for the date command, so lop off the first 10 digits and use that, so in the example we use 1261169238.

$ date -d @1261169238
Fri Dec 18 14:47:18 CST 2009

Not short and sweet, but it works.