Tag: Python

Delayed Queues for RQ

April 15, 2013 » Geek

I really like RQ. It’s got a sensible API and it’s built for Python, something I can’t say about Resque, pyres is fine, but it’s not the same.

My one beef with RQ is that I can’t delay a job for an arbitrary amount of time. You queue it up and it runs. To get around that, I built a simple delayed queue system for RQ, using the same redis backend.

My delayed queues leverage sorted sets to store and select jobs. We put the job in with it’s earliest run timestamp as the score, then we have a cron job or daemon that pulls them out when they are ready and pushes them over to RQ. Simple enough!

Here’s the really relevant code, everything else is trimming.

Delaying Jobs

Waking Jobs

You can run this at a granularity as low as one second the way it is written, and I’m sure you could go tighter if you wanted. We run it at a minute granularity, since our jobs are not highly time sensitive.

And because redis is atomic, you can run enqueue daemons/cron jobs on multiple machines, and everything should work fine, which is great for availability.

Grab the code and examples here if you are interested, https://gist.github.com/jmhobbs/5358101.

Thursday Quote: Tim Peters

June 21, 2012 » Geek, Life

“In the face of ambiguity, store refuse the temptation to guess.”

– Tim Peters
PEP 20

Tags: ,

Hot Patching Python Objects

June 4, 2012 » Geek

Hot patching is a touchy issue, it’s often frowned upon, and rightly so. But sometimes it’s the easiest way to get what you want.

Usually in Python you would patch a class with a new method, which works fine. But what if you only wanted to patch a single instance of that class, and not all of them?

Turns out that’s pretty easy with the types library.

To use it you define your patch method as a regular function, then you call types.MethodType to properly bind it, and you are done!

Here is a simple example:


Addressing Nested Dictionaries in Python

March 13, 2012 » Geek

I think that post title is right. Essentially I mean dynamically accessing attributes from nested dictionaries or tuples.

Let’s say you’ve got a JSON response like this:

The JSON module would convert that into nested dictionaries, like this:

In my case, I was trying to provide a runtime specified format string which could use any of the values at any depth.

There isn’t (to my knowledge) an easy way to address into a deep structure with a single string. I considered value based format strings ('%(name)s' but there is no way to descend from there either.

My solution was to use a dot notation and evaluate it for field values.

This requires a strict policy not to use dots in your keys, but that is not an issue for my use case.

Here is my code for the dot notation:

And here it is in use against the object above:

The next (optional) step would be to create a wrapper object.

Which we can then use like so:

While this is just sugar, it does look nice doesn’t it? To be complete you would want to implement the other sequence methods such as __setitem__

So that’s my fix – what’s yours?

Thursday Quote: Kenneth Reitz

February 16, 2012 » Geek

“The API is all that matters. Everything else is secondary.”

– Kenneth Reitz
Python for Humans

Tags: , , ,