What a named scope does in Rails, in Django terms, is allow you to name a given query and use it as a base, or “scope,” for further queries. In other words, your narrowing down the results upon which additional queries that you run will be executed.

It’s pretty much the same idea behind modifying initial Manager QuerySets as outlined in the Django documentation, with one major difference: Scopes can be chained. Chaining is the feature I had long coveted, and I had been banging my head around for a solution for the last few months.

The other day while I was poking around the internals a bit though, a decent idea came to me. I realized it would be quite simple for me to extend the default Manager class with an attribute where I could store QuerySet methods I wanted to execute as part of a scope, and then overwrite get_query_set() with a hook that ran these methods before executing the rest of the query.

This turned out to be a nice strategy, and with the addition of some decorators the interface itself turned out to be extraordinarily simple.

Anyhow, here’s a walkthrough of how my alpha “scopable” module would work, with example code.

Let’s start with a simplistic blog post model:

class Post(models.Model):
    title           = models.CharField('title', max_length=200)
    author          = models.ForeignKey(User)
    body            = models.TextField('author')
    publish_at      = models.DateTimeField('published at', default=datetime.now)
    is_draft        = models.BooleanField('is draft', default=False)
    is_deleted      = models.BooleanField('deleted', default=False)
    objects         = PostManager()

That’s enough to provide for some good example scopes. Now, here’s what PostManager might look like, with explanation to follow:

from scopable import ScopableManager, scope

class PostManager(ScopableManager):
    @property
    @scope
    def active(self):
       return self.filter(is_deleted=False, user__is_active=True)

    @property
    @scope
    def published(self):
       return self.filter(is_draft=False)

    @property
    @scope
    def public(self):
       return self.active.published

So as you can see, the ScopableManager class is very easy to use; you just add the @scope decorator to any method that you want to turn into a scope. (Note that I added the @property decorator as a stylistic preference; it’s not required.)

To use the code, you just have to be sure to call a QuerySet method after it, e.g.:

Post.objects.active.all()
Post.objects.published.all()
Post.objects.active.published.filter(publish_at__lte=datetime.now())
Post.objects.public.all()

The only difference between a scoped and unscoped method is that execution of any QuerySet methods defined in a scoped method are deferred until a QuerySet method is called (e.g. Post.objects.public.all()), and scoped methods return an instance of your manager back, so they can be chained. As you can see from the public method, scoped methods can be nested as well.

Why is this awesome? Well the main advantage from my perspective is that it allows your business rules to be defined in one and only one discrete place.

So if, for example, I added a new business rule that blog posts shouldn’t be considered published unless they were not drafts and the publish date had already passed, I could just add that rule to the published method:

    @property
    @scope
    def published(self):
       return self.filter(publish_at__lte=datetime.now(), is_draft=False)

This means that any other queries that were built on published, such as public, are now updated with the new business logic we added to published.

The implementation of all of this is pretty straightforward. The main complication comes from implementing scope as a decorator and trying to elegantly capture all of the QuerySet methods used in a scoped Manager method.

This is still alpha code, and as of now it doesn’t handle custom Manager methods that are not scoped. I mostly just wanted to get this idea out there before I do any more work:

"""
scopable.py - A module for providing Rails-like named scopes to Django model Managers
Verson 0.1
"""

from functools import wraps
from django.db import models

class DummyManager(object):
    def __init__(self, actual_mgr, calls=[]):
        super(DummyManager, self).__init__()
        self.actual_mgr = actual_mgr
        self.calls = calls

    def __call__(self, *args, **kwargs):
        last_call = self.calls[-1]
        last_call["is_property"] = False
        last_call["args"] = args
        last_call["kwargs"] = kwargs
        clone = DummyManager(self.actual_mgr, self.calls)
        self.calls = []
        return clone

    def __getattribute__(self, name):
        if name in ["calls", "actual_mgr"]:
            return super(DummyManager, self).__getattribute__(name)

        # attr_type, args, and kwargs are set here by default in case the attr never gets called
        call = { "attr": name, "is_property": True, "args": [], "kwargs": {} }

        # if this is a default manager method
        if hasattr(models.Manager, name):
            call["definition"] = "default"

        # else if this is a custom manager method
        elif hasattr(self.actual_mgr.__class__, name):
            call["definition"] = "custom"

        # if it can't be found in those two classes, throw an error
        else:
            raise AttributeError("'%s' object has no attribute '%s'" % (self.actual_mgr.__class__.__name__, name))

        self.calls.append(call)
        clone = DummyManager(self.actual_mgr, self.calls)
        self.calls = []
        return clone

def scope(fn):
    @wraps(fn)
    def scopified(*args, **kwargs):
        args_list = list(args)
        actual_mgr = args_list.pop(0)
        dummy = DummyManager(actual_mgr, [])
        args_list.insert(0, dummy)
        deferred_calls = get_deferred_calls(actual_mgr, fn, args_list, kwargs)
        a = actual_mgr.__class__(deferred_calls=deferred_calls)
        a.contribute_to_class(actual_mgr.model, actual_mgr.model.__name__)
        return a
    return scopified

def get_deferred_calls(mgr, fn, args, kwargs):
    deferred_calls = []
    result = fn(*args, **kwargs)
    if result.calls:
        for call in result.calls:
            if call["definition"] == "default":
                deferred_calls.append(call)
            elif call["definition"] == "custom":
                next_result = getattr(mgr, call["attr"])
                if not call["is_property"]:
                    next_result = next_result(*call["args"], **call["kwargs"])
                deferred_calls = deferred_calls + next_result.deferred_calls
            else:
                raise AssertionError("Something went wrong.")
    return deferred_calls

class ScopableManager(models.Manager):

    def __init__(self, deferred_calls=[]):
        super(ScopableManager, self).__init__()
        self.scopes = []
        self.deferred_calls = deferred_calls

    def get_query_set(self):
        qs = super(ScopableManager, self).get_query_set()
        if self.deferred_calls:
            for deferred_call in self.deferred_calls:
                next_qs = getattr(qs, deferred_call["attr"])
                if not deferred_call["is_property"]:
                    next_qs = next_qs(*deferred_call["args"], **deferred_call["kwargs"])
                qs = next_qs
        return qs

Please let me know in the comments if you have feedback, suggestions, bugs, grave warnings, etc.

Well, I hit some snags as I got underway. Partly this had to do with me being something of a novice when it comes to testing, and partly it has to do with the state of testing in Django at present.

My mini-gripe with testing in Django is that it’s just kind of loosey-goosey. You can use doctest, or you can use unittest. You can put your tests in your functions, or in your models, or in tests.py, or in a tests/ subfolder.

For me at least, I want my framework to be a bit more opinionated. I want a bunch of guys who are all smarter than me to have sat around and thought about (or better yet, generalized from a working production code) things like the best way to configure testing for a Python web framework, and then to tell me, in no uncertain terms, the exact way to do it.

I’m obviously exaggerating a bit, and I don’t really mean at all to rant about a phenomally awesome, not to mention open source, framework like Django. What all of the above amounts to is that I had to ponder deeply on the problem space myself and come up with a way of structuring my test code that would work for me.

So that’s what I did, and here is the organizational structure I intend to use as I proceed.

An Organizational Structure for Test Code in Django

This is how Wikipedia defines unit testing:

In computer programming, unit testing is a software design and development method where the programmer verifies that individual units of source code are working properly. A unit is the smallest testable part of an application.

One of the things that kept frustrating me about writing tests was that my tests were all methods of a test case class that referred to a model class or a view module. Most of the examples you see are written in that form, such as the following from the Django testing documentation:

class AnimalTestCase(unittest.TestCase):
    def setUp(self):
        self.lion = Animal.objects.create(name="lion", sound="roar")
        self.cat = Animal.objects.create(name="cat", sound="meow")

    def testSpeaking(self):
        self.assertEquals(self.lion.speak(), 'The lion says "roar"')
        self.assertEquals(self.cat.speak(), 'The cat says "meow"')

What frustated me was that you’d end up with a big long catalog of test methods for all the various methods of Animal. It would also be hard to determine what the test was referring to by looking at the test method name. Sometimes a method might be testSpeaking(); other times it might be testShouldSpeak(). In either case, I didn’t feel like the methods were serving very well as documentation, which is one of the supposed benefits of user testing.

Then as I re-read the definition of a unit test to myself for the hundredth time, it dawned on me. Unit tests are supposed to focus on *the smallest testable part of an application.* But all the examples I was seeing had tests that were organized around the class, which is certainly not the smallest testable part. That distinction belongs to methods and class attributes.

I stumbled upon my first rule: Organize test methods around the individual methods and attributes of a class, instead of the class itself. How so? The easiest is to create a separate test case subclass for *each* method and attribute. Each test method of that subclass is then a test of a behavior of the method or attribute under test.

This mini-breakthrough also helped me flesh out a rule for writing better test method names: Start every test method name with “test_should_”. This borrows from a page in behavior-driven development, but basically “should” is the perfect word to use to express a desired behavior.

In Practice

Here’s what all this looks like in practice, taking the Animal class as an example again from Django’s testing documentation:

# models.py
from django.db import models

class Animal(models.Model):
    name = models.CharField(max_length=20)
    sound = models.CharField(max_length=20)

    def speak(self):
        return 'The %s says "%s"' % (self.name, self.sound)

And here’s how I would propose that test code for this class be structured (with a focus on the organization, not thoroughness of testing or appropriateness of implementation):

from django.test import TestCase
from myapp.models import Animal

class AnimalTestCase(TestCase):
    def setUp(self):
        self.lion = Animal.objects.create(name="lion", sound="roar")

class Class(AnimalTestCase):
    # obviously a silly, trivial test
    def test_should_instantiate_object_of_its_own_type(self):
        self.failUnlessEqual(type(Animal()), Animal)

class NameAttribute(AnimalTestCase):
    def test_should_be_defined(self):
        self.failUnless(hasattr(self.lion, 'name'))

    def test_should_accept_string_value(self):
        a = Animal()
        a.name = "bear"
        a.save()
        self.failUnless(a.id) # object was saved successfully

    def test_should_allow_no_more_than_20_chars(self):
        self.lion.name = "a"*21
        self.failUnlessRaises(Warning, self.lion.save)

class SoundAttribute(AnimalTestCase):
    def test_should_be_defined(self):
        self.failUnless(hasattr(self.lion, 'sound'))

    # similar test_shoulds as with 'name' here

class SpeakMethod(AnimalTestCase):
    def test_should_be_defined(self):
        self.failUnless(hasattr(self.new_pp, 'speak'))

    def test_should_return_appropriate_string(self):
        self.failUnlessEquals(self.lion.speak(), 'The lion says "roar"')

(I have not executed the above code, so please don’t be surprised if it has errors.)

So, it’s pretty simple really. Each attribute or method being tested is given an explicitly named test class, which is a subclass of a single test case class for the class under test. Each specified behavior of that attribute or method is assigned a test method beneath the appropriate test class.

Here’s what I like about this approach:

• You don’t have to “think” while writing tests. If you’ve decided to define a method, you just list off to yourself all of the things it should do and/or require.
• The tests actually read like documentation. Not that you’ll ever see them expressed anywhere in this format, but I like to read them as “NamedAttribute.should_accept_test_value” or “SpeakMethod.should_return_appropriate_string”. It’s just so easy to make sense out of exactly what each part of your model is supposed to do.
• The class names provide some grouping structure to the test methods. Ten groups of ten methods, for example, are a lot easier to make sense of and keep organized than 100 test methods without any other structure.

Getting It To Work

It’s actually quite easy to get this all to work. The only trick involved is that you have to specify which suites to run to the django test runner. (This may not in fact be a requirement, but I wanted adding new test case subclasses to be quick and easy; I didn’t want to spend my life configuring which tests to run in a bunch of places.)

Here’s a quick function I wrote to handle this:

def build_test_suite_from(test_cases):
    """
    Returns a single or group of unittest test suite(s) that's ready to be
    run. The function expects a list of classes that are subclasses of
    TestCase.

    The function will search the module where each class resides and
    build a test suite from that class and all subclasses of it.
    """
    test_suites = []
    for test_case in test_cases:
        mod = __import__(test_case.__module__)
        components = test_case.__module__.split('.')
        for comp in components[1:]:
            mod = getattr(mod, comp)
        tests = []
        for item in mod.__dict__.values():
            if type(item) is type and issubclass(item, test_case):
                tests.append(item)
        test_suites.append(unittest.TestSuite(map(unittest.TestLoader().loadTestsFromTestCase, tests)))
    return unittest.TestSuite(test_suites)

Django allows you to define a suite() function in your test module, which is then automatically detected by the test runner when tests are executed. So we’ll just embed the above function in that, like this, for example:

from myapp.test_utils import build_test_suite_from
from myapp.tests.models.some_model import SomeModelTestCase
from myapp.tests.models.another_model import AnotherModelTestCase

test_cases = [
    SomeModelTestCase,
    AnotherModelTestCase
]

def suite():
    return build_test_suite_from(test_cases)

Here’s what happens: build_test_suite_from() runs through each TestCase you pass to it, and searches the module where that TestCase is found for subclasses of that TestCase (including the TestCase itself). It then builds a test suite containing all of the test methods out of each of these, and then combines all of them together into one giant test suite which Django will run automatically when you do ./manage.py test.

The idea again is to keep configuration to a minimum. For each model you create test cases for you have to import the model’s TestCase and add it to the test_cases list, but no additional configuration is required for subclasses you create of that TestCase.

I expect that this structure will evolve as I actually start using it more. In particular, tests for forms and views may or may not work as cleanly within this structure as do those for models.

If you have any comments or feedback (good or bad), I’d love to hear about it below.

It started off pretty easy — just use “an” when the first letter of the phrase is a vowel and otherwise use “a.” And then I remembered about abbreviations (“an HR department”), silent H words (“an honorable man”), and exceptions among a few O and U words (“a one-time offer” or “a union representative”), and suddenly it wasn’t so easy.

Fortunately I managed to find an excellent database of English words that includes pronunciations, and in no time at all I was able to generate a list of exceptions.

Here’s the final result, implemented as a custom django template tag (in my use case, I needed to able to wrap the original phrase in an HTML tag, so I couldn’t use a filter tag). This could also easily be a standalone Python function as well (literally by removing the @register.simple_tag decorator).

import re

@register.simple_tag
def with_indefinite_article(phrase, before_phrase="", after_phrase=""):
    silent_h_words = ["homage", "hour", "herb", "heir", "honest", "honor"]
    letters_with_initial_vowel_sound = ["a", "e", "f", "h", "i", "l", "m", "n", "o", "r", "s", "x"]
    vowels = ["a", "e", "i", "o", "u"]
    o_exceptions = ["once","one","oneness","ones","oneself","onetime"]
    u_exceptions = ["ubiquitous","uganda","ugandan","ukraine","ukrainian","ukulele","ukuleles","unanimous","unanimously","unicellular","unicorn","unicorns","unicycle","unicycles","unification","unified","uniform","uniformed","uniformly","uniforms","unify","unifying","unilateral","unilateralism","unilaterally","union","unionist","unionists","unionization","unionized","unionizing","unions","unique","uniquely","uniqueness","unisex","unison","unisons","unit","unitarian","unitary","unite","united","uniting","units","unity","universal","universally","universe","universes","universities","university","universitys","unix","uranium","urinalysis","urinary","urinate","urinating","urine","urologist","urologists","urology","uruguay","uruguayan","uruguays","usable","usage","usages","use","used","useful","usefully","usefulness","useless","usenet","user","users","uses","usual","usually","usurp","usurpation","usurped","usurper","usurping","usury","utah","utahn","utahns","utahs","utensil","utensils","uterine","utero","uterus","utilitarian","utilities","utility","utilitys","utilization","utilize","utilized","utilizes","utilizing","utopia","utopian","utopians","utopias"]

    # get rid of everything but letters and numbers and a few characters
    first_word = re.sub(r"[^A-Za-z0-9\-]", "", phrase.split()[0])
    first_letter = first_word[0].lower()

    # default to 'a'
    article = "a"

    if first_word.isupper() or (len(first_word) == 1):
        # if all caps or a single letter, we're going to assume it's an abbreviation
        if first_letter in letters_with_initial_vowel_sound:
            article = "an"
    elif first_letter == "h" and any( [re.match(r'(?i)%s' % w, first_word) for w in silent_h_words] ):
        article = "an"
    elif first_letter in vowels:
        if first_letter == 'e':
            article = "a" if first_word.lower().split('-')[0] == 'ewe' else "an"
        elif first_letter == 'o':
            article = "a" if first_word.lower().split('-')[0] in o_exceptions else "an"
        elif first_letter == 'u':
            article = "a" if first_word.lower().split('-')[0] in u_exceptions else "an"
        else:
            article = "an"

    return "%s %s%s%s" % (article, before_phrase, phrase, after_phrase)

Hopefully this will save someone who stumbles upon this some extra work down the road. If you find any bugs, please let me know in the comments.

In today’s example, the goal was pretty straightforward. We have a model class, Topic, that has a one-to-many relationship to a ThreadedComment model class. Each ThreadedComment for each Topic, has a User associated with it; furthermore the Topic itself as a User associated with it (the creator of the topic).

The issue here is that it’s not actually a straightforward relationship between Topic and ThreadedComment, due to some complicating factors beyond the scope of this discussion. Suffice it to say what we’re looking for is a list of users who are participants in the discussion, inclusive of the topic creator. But we’re not looking for just any old list — what we need is a QuerySet, for the reason that we are making use of some specialized tags in our templates that require data to be in a QuerySet.

Here’s a quick example of what we want the interface to look like in the template:

	<ul>
	{% for participant in topic.participants %}
		<li>{{ participant.name }}</li>
	{% endfor %}
	</ul>

So let’s do this step by step.

First, let’s get a list of the ThreadedComment objects we need:

ThreadedComment.public.all_for_object(topic))

The all_for_object() method is one of the “complications” we were referring to above. All we care about is that it returns a set of ThreadedComment objects for a given topic.

Okay, but now that we have this set of objects, how are we going to return a QuerySet that yields us User objects?

My co-worker actually showed this to me, so I’ll share it here. We’ll use in keyword argument in filter(), which maps to the SQL IN clause. The Django documentation has a simple example:

Entry.objects.filter(id__in=[1, 3, 4])

is equivalent to:

SELECT ... WHERE id IN (1, 3, 4)

So that’s just what we’re going to do: pass a list of user ids as an in keyword argument to identify the users we want.

First, let’s extend our original QuerySet to get the values we need as a list:

ThreadedComment.public.all_for_object(self).values_list('user_id', flat=True)

One thing is missing — we need to include the user id of the topic creator, as I mentioned above. We’ll just evaluate the QuerySet to a list and then add the topic creator to it:

list(ThreadedComment.public.all_for_object(self).values_list('user_id', flat=True)) + [self.creator.id]

Now let’s take the list of ids and pass it to a QuerySet that we’ll return User objects. Our list goes in the pk__in keyword argument:

User.objects.filter(
    pk__in=list(
        list(ThreadedComment.public.all_for_object(self).values_list('user_id', flat=True)) + [self.creator.id]
    )
)

Now let’s add this as a method participants() to the Topic model and throw a @property decorator on it so we can call it as an attribute:

@property
def participants(self):
    return User.objects.filter(
        pk__in=list(
            list(ThreadedComment.public.all_for_object(self).values_list('user_id', flat=True)) + [self.creator.id]
        )
    )

And that’s all there is to it.

What I love about this approach is that it’s a pattern that fits in to so many places (so many that I almost feel like it deserves a higher level abstraction within the QuerySet API). There are variations that make it slightly more efficient as well; for example, if I didn’t need to append the topic creator id in my example, then I could have passed the list of ids as a query as opposed to evaluating it:

User.objects.filter(
    pk__in=ThreadedComment.public.all_for_object(self).values_list('user_id', flat=True)).query
)

The advantage here is that this is evaluated as a subselect statement, so it only requires a single database call.

Anyhow, this technique is really fantastic when you have a set of related objects that for whatever reason you aren’t able to easily obtain as a QuerySet. Enjoy.

I was pretty dissatisified with my brief survey of existing chat widgets out there. This is a professional conference, so I really didn’t want a widget that was all yucked up with ads or smileys.

Anyhow, since I use Campfire a lot at my day job, I figured, why not give it a shot?

The good news is, Campfire does have an API, known as Tinder.

The bad news is, it’s an unoffical API, written in Ruby. (Not that there’s anything wrong with that, but I didn’t want to deal with configuring a reliable production environment for what was hopefully a small project.)

The good news is, someone ported the Tinder API over to Python, calling it Pinder. Which rocks, because now I get to use Django.

To make a long story short, here’s what I ended up doing:

• Set up a custom django-admin command to execute the listener. So now I just run $ python manage.py listen_to_campfire at the command prompt and away it goes.
• Created a listener class that implements the Pinder API and that basically continually loops over the transcript and adds any new messages it finds to the DB. Here’s the main function:

    def listening_to_campfire(self):
        """
        The actual listening method. This method checks for new campfire messages and writes any it finds
        to the Message model. It returns True if everything goes okay, and False if any errors get caught.
        """

        try:
            current_messages = self.room.transcript(self.room.transcripts()[0])
            most_current_message_id = int(current_messages[-1]['id'])
            if not most_current_message_id == self.last_message_id: # i.e. there are new messages

                # compares the id of each new message with the latest existing message id,
                # only keeps those that are newer
                new_messages = filter(
                    lambda cm: int(cm['id']) > self.last_message_id,
                    current_messages
                )

                for nm in new_messages:
                        m = Message.create_from_campfire_message(nm)
                        if m:
                            m.save()
                            print m

                self.last_message_id = most_current_message_id
            else:
                print "..."
        except KeyboardInterrupt:
            script_halted()
        except:
            return False

        return True

• Created a view of the latest messages from the database and returned them in JSON.
• Implemented an Ajax routine to poll the JSON feed and add any new results to the chat widget in real time.

This works pretty nicely. The only thing the chat widget does not allow at present is the ability to type messages from the widget itself…for that you have to enter the Campfire chat room. This may turn out to be a important requirement though, in which case I will need to extend the Pinder API to allow for anonymous room logins.

To be continued…perhaps.

UPDATE: The yucks won. I’m using a Meebo room, which I managed to format decently though.