Sneaker-netting Huge Files with split and cat

Big files happen, and sometimes they need to be moved.

I’ve found that moving big files between local computers is often fastest with a USB drive and my shoes, especially since I don’t really have access to a wired network usually.

But sometimes, files are just too big for your thumb drive. Some of us don’t carry huge drives around, I usually only have a spare 2GB on me.

Suppose you have a 4.4GB ZIP file (don’t ask, just suppose). I’ve only got a 2GB (1.9GB usable) thumb drive on me, but I need to move it over to another machine and I don’t have all day.

In the past I’ve use dd tricks, but I knew there had to be a better way.

Enter split and cat. cat concatenates files, and it’s cousin split, well, splits them.

Perfect! Just what I needed! I’ll split my file into chunks with split, sneaker it over the other machine and cat it all back together.

Split

computer-one $ split -b 1500M big.zip
(...wait...)
computer-one $ ls -l
-rw-rw-r--  1 jmhobbs jmhobbs 1572864000 2012-02-27 14:44 xaa
-rw-rw-r--  1 jmhobbs jmhobbs 1572864000 2012-02-27 14:45 xab
-rw-rw-r--  1 jmhobbs jmhobbs 1572864000 2012-02-27 14:46 xac
-rw-rw-r--  1 jmhobbs jmhobbs    7637844 2012-02-27 14:46 xad
computer-one $

Move

Join

computer-two $ cat xaa xab xac xad > big.zip
(...wait...)
computer-two $

Enjoy!

computer-two $ ls -l
-rw-rw-r--  1 jmhobbs jmhobbs 4726229844 2012-02-27 15:02 big.zip

Developing For Twilio In PHP – Introducing Twillip

Last weekend I had the fun opportunity of building my first Twillio app with a group at the Omaha Startup Weekend. We wrote a phone based version of chattroulette called Call Spinner which includes a cool advertising system we hope to expand on.

Developing for Twilio isn’t that hard, it’s mostly just about generating the right XML. We used the PHP library Twilio provides and it’s pretty nice, but debugging off of the Twilio website was a waste of money and not much fun.

So, over the last few days I worked out a nice debugging tool I decided to call Twillip. What it does is conditionally wrap your code and prints it in a clean fashion, with links and variables and other goodies like that.

Super simple to use too! Here, this is a contrived example. Normally you probably want some sort of cool dispatcher (I have one of those if you want it) and to check that requests are signed, etc.

example.php

addSay( 'This app uses Twillip for obviously awesome reasons!' );
    $r->addPlay( 'funky-beats.mp3', array( 'loop' => 3 ) );
    $r->addRedirect( '/doesntexist.php' );
  }
  else {
    $r->addSay( 'Oh no! I didn\'t get sent a phone number! Who in blue blazes are you?' );
    $r->addSay( 'This line will generate a PHP warning now: ' . $_REQUEST['Caller'] );
  }
  $r->respond();

  if( IS_DEV ) { Twillip::End(); }

Sweet, so what does that buy us? Only this awesome interface is all!

But there is an error! $_REQUEST['Caller'] isn’t set. We can fix that by clicking on the Add New Input and filling it in.

There it is! Now we can click Reload Page and it will refresh with all of the inputs.

You can’t tell in the image, but the Redirect and Play verbs are both clickable links, and will bring all of the variables in the input section with them.

Seeing is believing, so try it out at http://static.velvetcache.org/pages/2010/09/16/developing-for-twilio-in-php-introducing-twillip/demo/

You can grab the source at http://github.com/jmhobbs/Twillip.

Auto-Generated Github User Page With py-github

Github has a cool feature called “Github Pages” that let you host static content on a subdomain of github, e.g. http://jmhobbs.github.com/.

They also provide an auto-generator for project pages that has a nice clean format which I really like. So I decided to make my user page match the look and feel of the project pages. And to boot I wanted to be able have it auto-generate since I want it to be “hands free”, otherwise I’ll forget to update it.

To make this happen I whipped up my template and then grabbed the excellent py-github from Dustin Sallings, which I have used before.

Without furthur ado I’ll just show you the source. It’s not complicated, just some API calls then search replace on a template file. If you want to use it, be sure to get the most recent version from http://github.com/jmhobbs/jmhobbs.github.com.

Throw in a cron job and you are set. Beware of lot’s of “page build” notices from Github though.

# -*- coding: utf-8 -*-

import github.github as github
import yaml
import time
from datetime import datetime

def repo_date_to_epoch ( date ):
  epoch = time.mktime(
    time.strptime(
      date[0:-6],
      "%Y-%m-%dT%H:%M:%S"
    )
  )
  return int( epoch )

def main ():

  print "Loading settings...."
  f = open( 'settings.yaml' )
  settings = yaml.load( f )
  f.close()

  gh = github.GitHub()

  print "Fetching user information..."
  user = gh.users.show( settings['username'] )

  print "Fetching repository information..."
  repos = gh.repos.forUser( settings['username'] )

  print "Sorting repositories..."
  repos = sorted( repos, cmp=lambda a, b: repo_date_to_epoch( b.pushed_at ) - repo_date_to_epoch( a.pushed_at ) )

  print "Loading template..."
  f = open( 'index.html.tpl' )
  template = f.read()
  f.close()

  print "Mangling template..."
  template = template.replace( '<% username %>', settings['username'] )
  template = template.replace( '<% fullname %>', user.name )
  template = template.replace( '<% email %>', user.email )
  template = template.replace( '<% following %>', str( user.following_count ) )
  template = template.replace( '<% followers %>', str( user.followers_count ) )
  template = template.replace( '<% publicrepos %>', str( user.public_repo_count ) )

  repo_string = ''

  for repo in repos:
    if repo.private:
      continue

    repo_string = repo_string + '
' + repo.name + ''

    try:
      repo_string = repo_string + ' - ' + repo.homepage + ''
    except AttributeError:
      pass

    repo_string = repo_string + '
'

    repo_string = repo_string + "Forks: " + str( repo.forks ) + " - Watchers: " + str( repo.watchers ) + ' | '

    if repo.has_issues:
      repo_string = repo_string + ' Issues |'

    if repo.has_wiki:
      repo_string = repo_string + ' Wiki |'

    if repo.has_downloads:
      repo_string = repo_string + ' Downloads |'

    repo_string = repo_string + '
Last Push: ' + datetime.fromtimestamp( repo_date_to_epoch( repo.pushed_at ) ).ctime()

    try:
      repo_string = repo_string + '
' + repo.description + '< /pre>'
    except AttributeError:
      repo_string = repo_string + '

'
      pass

    repo_string = repo_string + "
\n"

  template = template.replace( '<% repos %>', repo_string )

  ga = """
    
  """

  if False != settings['google_analytics']:
    template = template.replace( '<% google_analytics %>', ga )
    template = template.replace( '<% ga_code %>', settings['google_analytics'] )
  else:
    template = template.replace( '<% google_analytics %>', '' )

  print "Writing file..."
  f = open( 'index.html', 'w' )
  f.write( template )
  f.close()

  print "Done!"

if __name__ == "__main__":
  main()

Wow. You actually scrolled through all of that. Amazing.