Tag: Tutorial

Writing a Plugin for Storytlr

January 19, 2010 » Geek

Update: 2010-02-18
I put this onto the Storytlr wiki a while back. I highly recommend viewing that version instead of this one. This version will not be updated further.


Update: 2010-01-19
Edited in a minor fix, I learned a bit more when I wrote my foursquare plugin.
The correct revision to follow along with is now 75c520df.

I recently got interested in the lifestream application Storytlr.

One of the first things I wanted to do was add a plugin for github, as that is a fairly large part of my online life. Unfortunately there is very little documentation, which is understandable for a private project that just went open.

As such I had to work through some code, but all in all it wasn’t too tough. The source is very readable, just not set up with any auto-documentation.

So, as a way of giving back, I’m going to walk through the Storytlr plugin system, as I understand it. If you want to play along at home, you can grab the source from my github, and the commit we will be working from is 23136196

First things first, we need to copy off another plugin, that’s the easy way to get started. I cloned the RSS one, but they are all pretty similar to start with. Here’s what my final file tree looks like, it’s pretty similar to the starting tree, just with “Github” instead of “RSS”. This would reside at path_to_app/protected/application/plugins/github

├── database.sql
├── github.png
├── models
│   ├── GithubItem.php
│   └── GithubModel.php
└── views
    └── scripts
        ├── rss.phtml
        ├── story.phtml
        └── timeline.phtml

So what are each of these and what do we have to do to them? Well, I like to drive my development by my data structures, so let’s start with database.sql. This is pretty straightforward SQL, and I’ll highlight some key fields after you have a look.


DROP TABLE IF EXISTS `github_data`;
SET @saved_cs_client = @@character_set_client;
SET character_set_client = utf8;
CREATE TABLE `github_data` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `source_id` int(10) unsigned NOT NULL,
  `github_id` varchar(255) NOT NULL,
  `title` text NOT NULL,
  `content` text,
  `repository` text,
  `link` varchar(255) NOT NULL,
  `published` varchar(45) NOT NULL,
  UNIQUE KEY `DUPLICATES` USING BTREE (`source_id`, `github_id`),
  FULLTEXT KEY `SEARCH` (`content`)
SET character_set_client = @saved_cs_client;

So what are our key items to look at here? Well, how about table naming convention? The existing tables are named as their plugin name, all lowercase, then _data. Why get fancy? Let’s use the convention.

What else is important? I kept the id and source_id pieces intact. These are actually fairly important, as I learned when I tried to make source_id into a varchar. source_id is actually the key that is tied to the sources table. That seems obvious now, but trust me, it didn’t when I was writing this the first time.

Aside from that, there isn’t too much to worry about. Just make sure to add something that you can use to identify individual updates on, github_id in this case, and then add your data fields and set your indexes. The naming of the FULLTEXT "SEARCH" key is probably important, though I didn’t test it any other way. You’ll probably find that all of your plugin tables will look more or less alike.

Now what? Well, let’s go ahead and define the model for an item, since we are already thinking about the data types. Here is a partial listing of GithubItem.php, have a look.

class GithubItem extends SourceItem {

  protected $_prefix   = 'github';

  protected $_preamble = 'Github activity: ';

  public function getContent() { return $this->_data['content']; }

  public function getTitle() {
    $title = str_replace(
      '' . $this->_data['repository'] . '',
      html_entity_decode( strip_tags( $this->_data['title'] ) )
    return $title;

  public function getLink() { return $this->_data['link']; }

  public function getType() { return SourceItem::LINK_TYPE; }

  public function getBackup() {
    $item = array();
    $item['SourceID'] = $this->_data['source_id'];
    $item['Title'] = $this->_data['title'];
    $item['Content'] = $this->_data['content'];
    $item['Repository'] = $this->_data['repository'];
    $item['Link'] = $this->_data['link'];
    $item['Published'] = $this->_data['published'];
    return $item;


This is a very stripped down implementation. All of your data is available to the views through the inherited SourceItem::toArray method, so you really only need to override a few methods and put in any mangle logic, as I did in the getTitle method.

One other method I was sure to override was getType. The types are enumerated in /protected/application/admin/models/SourceItem.php, and are as follows:

  const IMAGE_TYPE  = 'image';

  const AUDIO_TYPE  = 'audio';

  const VIDEO_TYPE  = 'video';

  const STATUS_TYPE   = 'status';

  const BLOG_TYPE   = 'blog';

  const LINK_TYPE   = 'link';

  const OTHER_TYPE  = 'other';

  const STORY_TYPE  = 'story';

I am not sure how they play into the rendering process at this point, but better safe than sorry, right?

Let’s move on to the views. These are all very similar, and are only rendered in different, well, views. I’ll go over timeline.phtml, but if you copy the others you should be able to piece it together in a jiffy.

item->getTitle(); ?>
item->getContent(); ?>

Pretty brutal, huh? You just get your title (mangled by GithubItem::getTitle in this case) and get your content, and you are as good as done.

Finally, I’m going to address the engine that drives all of this, GithubModel.php. A lot of this is just editing boilerplate, so let’s start with that. Comments are added in-line, but aren’t in the git source.

class GithubModel extends SourceModel {
  // What is the table named?
  protected $_name   = 'github_data';
  // What is the plugin directory named?
  protected $_prefix = 'github';
  // What fields are searchable? This is comma delimited, i.e. "content, title"
  protected $_search  = 'content';
  // What is a format string for update tweets?
  protected $_update_tweet = "Did %d things at github.com on my lifestream %s";
  // What is the service name shown on the backend/widgets
  public function getServiceName() {
    return "Github";
  // ?
  public function isStoryElement() {
    return true;
  // What is the URL for the widget links?
  public function getServiceURL() {
    return 'http://github.com/' . $this->getProperty('username');
  // Brief description for the admin interface
  public function getServiceDescription() {
    return "Github is social coding.";
  // What is the name on the account (for the admin interface mostly)
  public function getAccountName() {
    if ($name = $this->getProperty('username')) {
      return $name;
    else {
      return false;
  // What is the title of the account (for the front end)
  public function getTitle() {
    return $this->getServiceName();
  // The initial data import function
  public function importData() {
    $items = $this->updateData();
    $this->setImported( true );
    return $items;

Okay, now we are ready to dig into the meaty part, the data update functions. This is split into two parts, updateData and processItems. Here is a (hopefully) self-explanatory updateData implementation, more or less cloned from existing plugins.

  public function updateData() {
    $url  = 'http://github.com/' . $this->getProperty('username') . '.atom';

    $curl = curl_init();
    curl_setopt($curl, CURLOPT_URL, $url);
    curl_setopt($curl, CURLOPT_HEADER, false);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($curl, CURLOPT_USERAGENT,'Storytlr/1.0');

    $response = curl_exec($curl);
    $http_code = curl_getinfo($curl, CURLINFO_HTTP_CODE);
    curl_close ($curl);

    if ($http_code != 200) {
      throw new Stuffpress_Exception( "Github returned http status $http_code for url: $url", $http_code );

    if (!($items = simplexml_load_string($response))) {
      throw new Stuffpress_Exception( "Github did not return any result", 0 );

    if ( count( $items->entry ) == 0 ) { return; }

    $items = $this->processItems($items->entry);
    return $items;

Pretty easy, right? Just acquire the content (any way you wish, this one uses cURL), parse it out into an array and pass it on. If all goes well in processItems, then mark it as a successful update with SourceModel::markUpdated. Let’s see what processItems does.

  private function processItems($items) {
    $result = array();
    foreach ($items as $item) {
      $data = array();
      $data['title'] = $item->title;
      $data['repository'] = substr( $item->title, strrpos( $item->title, ' ' ) + 1 );
      $data['published'] = strtotime( $item->published );
      $data['content'] = $item->content;
      $data['link'] = $item->link['href'];
      $data['github_id'] = $item->id;
      $id = $this->addItem( $data, $data['published'], SourceItem::LINK_TYPE, array( $data['repository'] ), false, false, $data['title'] );
      if ($id) $result[] = $id;
    return $result;

Again, pretty simple. We take the passed in items, mangle them as needed and stuff them into a data array for storage. About the only unknown there is the addItem call, which is in the SourceModel class. Let’s take a look at the first part of that to understand our parameters.


  public function addItem($data, $timestamp, $type, $tags=false, $location=false, $hidden=false, $title=false) {
    $data['source_id']   = $this->_source['id'];
    $columns         = array();
    $keys            = array();
    $timestamp       = ($timestamp>=0) ? $timestamp : 0;

    foreach($data as $k => $v) {
      if (!$v) continue;
      $columns[] = "$k";
      $keys[] = ":$k";
      $data[":$k"] = "$v";

    $sql = "INSERT IGNORE INTO {$this->_name} (".implode(',', $columns).") "
       . "VALUES(".implode(',', $keys).")";

The most important thing for us to note are the names of the arguments: $data, $timestamp, $type, $tags, $location, $hidden, $title. These are self-explanatory and help us understand why the existing plugins pass what they do. Some other pieces to note is the override of source_id on line 169, and how it builds the query from your $data arguments, on lines 174-183. Naming matters!

So, now we are back to GithubModel, and we just have a few more methods to go. What remains below are the form generators and processing for the admin interface. Github only needs one piece of information to work, the username, so that’s all we are asking for below.


  public function getConfigForm($populate=false) {
    $form = new Stuffpress_Form();

    // Add the username element
    $element = $form->createElement('text', 'username', array('label' => 'Username', 'decorators' => $form->elementDecorators));

    // Populate
    if($populate) {
      $values  = $this->getProperties();

    return $form;

  public function processConfigForm($form) {
    $values = $form->getValues();
    $update  = false;

    if($values['username'] != $this->getProperty('username')) {
      $this->_properties->setProperty('username',   $values['username']);
      $update = true;

    return $update;

getConfigForm creates the form fields using the Stuffpress_Form class, which is described in /protected/library/Stuffpress/Form.php. Github only has the one element, so it is added and set to required, then we are done.

processConfigForm is similarly simple, we get the value of our field, then make sure it is valid. If it is, we save it into our model’s properties, which is a SourcesProperties class, which is in turn a Stuffpress_Db_Properties class. Essentially think of it as a persistent key/value store. Or don’t think about it at all, just use it.

At this point you should have a working plugin!


Debugging Storytlr can be tough sometimes, so make sure your config file has debug = 1, and keep an eye on /protected/logs/.

If you have any questions, comments or corrections, please let me know!

Extracting When You Visited A Page From Firefox

December 18, 2009 » Geek

Need to get the exact time that you visited a page in Firefox? I couldn’t find an easy way to look this up in the History interface, or anywhere else for that matter. I did however know that Firefox stores this kind of thing in sqlite3 databases. Here’s how I got what I needed.

First you have to find the sqlite databases, I’m on Linux so that would be in my home directory. The database you want is places.sqlite. Crack that open in sqlite3. Your command will differ as this is based on your profile name, mine is “gmail” so I ended up with g69ap5lc.gmail.

$ sqlite3 ~/.mozilla/firefox/g69ap5lc.gmail/places.sqlite

Be aware you have to shut down the Firefox instance first, because it locks the file. Make sure your privacy settings won’t erase it all when you shut it down! I had to change mine to “Remember history” first.

Next you need to find and grab the timestamp. This can be a chore if you don’t have the full URL. I was looking for the one from spiffie.org below.

sqlite>.headers on
sqlite>select * from moz_places;
1366|http://spiffie.org/kits/usb7/driver_linux.shtml|Linux USB7 Driver|gro.eiffips.|1|0|0||100|1261169238197827

The column we are interested in is last_visit_date which is 1261169238197827 in our case. You can also list all the recent visits from the moz_historyvisits table with the id column.

sqlite> select * from moz_historyvisits where place_id = '1366';

Now we need to convert that timestamp into something we can read (unless you are a super UNIX geek and can read timestamps). This instance is too precise for the date command, so lop off the first 10 digits and use that, so in the example we use 1261169238.

$ date -d @1261169238
Fri Dec 18 14:47:18 CST 2009

Not short and sweet, but it works.

Getting scripts and shortcuts to show in KDE4 Runner

December 8, 2009 » Geek

I tend to write little tidbits that I like to use in KDE fairly often, or custom shortcuts for launching applications. Every time I do this though, I forget how I got the last one to show up in krunner. This is just a quick guide to what I do, to remind myself and to show you.

1. Move script to somewhere in my path (I use ~/System/bin for no apparent reason)
2. Create a .desktop file
3. Move .desktop file to ~/.kde/share/applnk/
4. Run kbuildsycoca4 to rebuild the KDE4 cache
5. Restart krunner.

I also came across kappfinder, pictured below, that finds applications that aren’t in the menu system for some reason (only added to Gnome or something). Not useful for this application, but handy to know, and related.


Tags: ,

Hard subs with mencoder, or, annotating screencasts for free.

December 2, 2009 » Geek

I was working on a screen cast for a project and I could not get anything to work the way I wanted to. I didn’t have the desire to purchase software, so I sought a way to annotate it for free. I tried a half dozen free and open source video editors, with no real luck. Either they choked on the format I captured (from CamStudio) or they didn’t have a readily available inline text tool.

What I ended up using was plain old subtitles, plus the handyman’s secret weapon, mplayer/mencoder.

Here’s what I did. I fired up the video in VLC and found the points where I wanted to put my subtitle help text. Subtitle files are usually pretty straight forward. I chose the .srt format, which is plain text. You can edit by hand, or you can use a tool, I used the aptly named subtitleeditor. Which, by the way, barfed on my video file.

Here’s a snippet of the resulting .srt file:

00:00:00,000 --> 00:00:13,000
Welcome to the product gallery walk through.

00:00:13,000 --> 00:00:23,000
Adding a product: Select image.

00:00:23,000 --> 00:00:32,000
Insert title and description.

Next I needed to turn those “soft” subs into “hard” ones. This is where mplayer/mencoder come in. To add soft subs to a video in mplayer, you use the -sub option. Running this in mplayer first is a good way to see how it will look at tweak it according to the many options you have. I went with the defaults, it looks pretty good that way.

Lastly, I needed a good mencoder recipe to pull it all together. After lots of searching I found a great MPEG4 one here.

It’s a two pass system, and the crucial piece is in calculating the bitrate:

bitrate = 50 * 25 * width_of_video * height_of_video / 256

My bitrate was 2540600. After you get that, you just plug it into the two passes below. This includes my subtitle options.

mencoder -o /dev/null -ovc lavc -lavcopts vcodec=msmpeg4v2:vpass=1:vbitrate=2540600:mbd=2:keyint=132:vqblur=1.0:cmp=2:subcmp=2:dia=2:mv0:last_pred=3 -nosound -sub MySubTitles.srt -subfont-text-scale 3 RawScreenCast.avi
mencoder -o FinishedScreenCast.avi -ovc lavc -lavcopts vcodec=msmpeg4v2:vpass=2:vbitrate=2540600:mbd=2:keyint=132:vqblur=1.0:cmp=2:subcmp=2:dia=2:mv0:last_pred=3 -nosound -sub MySubTitles.srt -subfont-text-scale 3 RawScreenCast.avi

It takes a while but not too long. When all was said and done I had a perfect hard-subbed version and it shrank my file size from ~550MB to 7.6MB. That is a great encoding recipe.

Average video still.
An average frame, with hard subs.

Command Line Package Installation in OpenSuSE

October 1, 2007 » Geek

One of the things I love the most about Debian is apt. It’s a great and speedy package manager. Being able to apt-get install from the command line and not have to wait for a heavy UI to come up is a major plus for me. Thats why I was frustrated with OpenSuSE, which we use at work.

First thing first, I think that the OpenSuSE package system is a pig. Yast is a pig. Zypper is a pig. I did, however, find the fastest route to installing via the command line, which I’ll share here.

Unless you happen to know the exact name of the package you want to install, you’ll need to look it up. The fastest way I’ve found is using Webpin, a nice online package searcher. Be careful that you are reading from the right repo though. For example, a search on “magick++” returns many packages, including “libMagick++-devel (” which, to my Debian eye, looks like the perfect package. It is from an odd repo though, “Results from http://download.opensuse.org/repositories/home:/dipe/openSUSE_10.2” and the one I really want, and have access to mind you, is “ImageMagick-Magick++-devel (” which is in the main repo “Results from http://download.opensuse.org/distribution/10.2/repo/oss/suse”. It’s already too complicated, but we soldier on.

With my new, exact, package name in hand, I open up a root command line. The syntax for a zypper install is zypper install [package name] so I do zypper install ImageMagick-Magick++-devel and let it rip. After a ridiculous amount of parsing, it figures out the dependencies and asks me to continue, which I do. You have to babysit it for key managing, I havent figured out how to force “yes” on it. And that is how you use zypper to install on OpenSuSE.

root:~$ zypper install ImageMagick-Magick++-devel

Restoring system sources...

Parsing metadata for 20070918-142944...

Parsing metadata for 20070927-100843...

Parsing metadata for 20070927-100709...

Parsing metadata for 20070918-055437...

Parsing metadata for SUSE-Linux-10.2-Updates...

Parsing RPM database...









Continue? [y/n] y

Downloading: [S4:1][package]liblcms-devel-1.15-30.i586, 141.5 K(490.1 K unpacked)

Installing: [S4:1][package]liblcms-devel-1.15-30.i586

Downloading: [S4:1][package]libwmf-gnome-, 7.7 K(9.6 K unpacked)

Installing: [S4:1][package]libwmf-gnome-

Downloading: [S4:1][package]readline-devel-5.1-55.i586, 137.2 K(376.7 K unpacked)

Installing: [S4:1][package]readline-devel-5.1-55.i586

Downloading: [S4:1][package]libwmf-devel-, 414.1 K(2.7 M unpacked)

Installing: [S4:1][package]libwmf-devel-

Downloading: [S3:0][package]libxml2-devel-2.6.26-27.pm.1.i586, 2.5 M(11.6 M unpacked)

Installing: [S3:0][package]libxml2-devel-2.6.26-27.pm.1.i586

Downloading: [S5:0][package]ImageMagick-devel-, 1.5 M(5.6 M unpacked)

Installing: [S5:0][package]ImageMagick-devel-

Downloading: [S5:0][package]ImageMagick-Magick++-devel-, 193.2 K(939.8 K unpacked)

Installing: [S5:0][package]ImageMagick-Magick++-devel-