What a named scope does in Rails, in Django terms, is allow you to name a given query and use it as a base, or “scope,” for further queries. In other words, your narrowing down the results upon which additional queries that you run will be executed.

It’s pretty much the same idea behind modifying initial Manager QuerySets as outlined in the Django documentation, with one major difference: Scopes can be chained. Chaining is the feature I had long coveted, and I had been banging my head around for a solution for the last few months.

The other day while I was poking around the internals a bit though, a decent idea came to me. I realized it would be quite simple for me to extend the default Manager class with an attribute where I could store QuerySet methods I wanted to execute as part of a scope, and then overwrite get_query_set() with a hook that ran these methods before executing the rest of the query.

This turned out to be a nice strategy, and with the addition of some decorators the interface itself turned out to be extraordinarily simple.

Anyhow, here’s a walkthrough of how my alpha “scopable” module would work, with example code.

Let’s start with a simplistic blog post model:

class Post(models.Model):
    title           = models.CharField('title', max_length=200)
    author          = models.ForeignKey(User)
    body            = models.TextField('author')
    publish_at      = models.DateTimeField('published at', default=datetime.now)
    is_draft        = models.BooleanField('is draft', default=False)
    is_deleted      = models.BooleanField('deleted', default=False)
    objects         = PostManager()

That’s enough to provide for some good example scopes. Now, here’s what PostManager might look like, with explanation to follow:

from scopable import ScopableManager, scope

class PostManager(ScopableManager):
    @property
    @scope
    def active(self):
       return self.filter(is_deleted=False, user__is_active=True)

    @property
    @scope
    def published(self):
       return self.filter(is_draft=False)

    @property
    @scope
    def public(self):
       return self.active.published

So as you can see, the ScopableManager class is very easy to use; you just add the @scope decorator to any method that you want to turn into a scope. (Note that I added the @property decorator as a stylistic preference; it’s not required.)

To use the code, you just have to be sure to call a QuerySet method after it, e.g.:

Post.objects.active.all()
Post.objects.published.all()
Post.objects.active.published.filter(publish_at__lte=datetime.now())
Post.objects.public.all()

The only difference between a scoped and unscoped method is that execution of any QuerySet methods defined in a scoped method are deferred until a QuerySet method is called (e.g. Post.objects.public.all()), and scoped methods return an instance of your manager back, so they can be chained. As you can see from the public method, scoped methods can be nested as well.

Why is this awesome? Well the main advantage from my perspective is that it allows your business rules to be defined in one and only one discrete place.

So if, for example, I added a new business rule that blog posts shouldn’t be considered published unless they were not drafts and the publish date had already passed, I could just add that rule to the published method:

    @property
    @scope
    def published(self):
       return self.filter(publish_at__lte=datetime.now(), is_draft=False)

This means that any other queries that were built on published, such as public, are now updated with the new business logic we added to published.

The implementation of all of this is pretty straightforward. The main complication comes from implementing scope as a decorator and trying to elegantly capture all of the QuerySet methods used in a scoped Manager method.

This is still alpha code, and as of now it doesn’t handle custom Manager methods that are not scoped. I mostly just wanted to get this idea out there before I do any more work:

"""
scopable.py - A module for providing Rails-like named scopes to Django model Managers
Verson 0.1
"""

from functools import wraps
from django.db import models

class DummyManager(object):
    def __init__(self, actual_mgr, calls=[]):
        super(DummyManager, self).__init__()
        self.actual_mgr = actual_mgr
        self.calls = calls

    def __call__(self, *args, **kwargs):
        last_call = self.calls[-1]
        last_call["is_property"] = False
        last_call["args"] = args
        last_call["kwargs"] = kwargs
        clone = DummyManager(self.actual_mgr, self.calls)
        self.calls = []
        return clone

    def __getattribute__(self, name):
        if name in ["calls", "actual_mgr"]:
            return super(DummyManager, self).__getattribute__(name)

        # attr_type, args, and kwargs are set here by default in case the attr never gets called
        call = { "attr": name, "is_property": True, "args": [], "kwargs": {} }

        # if this is a default manager method
        if hasattr(models.Manager, name):
            call["definition"] = "default"

        # else if this is a custom manager method
        elif hasattr(self.actual_mgr.__class__, name):
            call["definition"] = "custom"

        # if it can't be found in those two classes, throw an error
        else:
            raise AttributeError("'%s' object has no attribute '%s'" % (self.actual_mgr.__class__.__name__, name))

        self.calls.append(call)
        clone = DummyManager(self.actual_mgr, self.calls)
        self.calls = []
        return clone

def scope(fn):
    @wraps(fn)
    def scopified(*args, **kwargs):
        args_list = list(args)
        actual_mgr = args_list.pop(0)
        dummy = DummyManager(actual_mgr, [])
        args_list.insert(0, dummy)
        deferred_calls = get_deferred_calls(actual_mgr, fn, args_list, kwargs)
        a = actual_mgr.__class__(deferred_calls=deferred_calls)
        a.contribute_to_class(actual_mgr.model, actual_mgr.model.__name__)
        return a
    return scopified

def get_deferred_calls(mgr, fn, args, kwargs):
    deferred_calls = []
    result = fn(*args, **kwargs)
    if result.calls:
        for call in result.calls:
            if call["definition"] == "default":
                deferred_calls.append(call)
            elif call["definition"] == "custom":
                next_result = getattr(mgr, call["attr"])
                if not call["is_property"]:
                    next_result = next_result(*call["args"], **call["kwargs"])
                deferred_calls = deferred_calls + next_result.deferred_calls
            else:
                raise AssertionError("Something went wrong.")
    return deferred_calls

class ScopableManager(models.Manager):

    def __init__(self, deferred_calls=[]):
        super(ScopableManager, self).__init__()
        self.scopes = []
        self.deferred_calls = deferred_calls

    def get_query_set(self):
        qs = super(ScopableManager, self).get_query_set()
        if self.deferred_calls:
            for deferred_call in self.deferred_calls:
                next_qs = getattr(qs, deferred_call["attr"])
                if not deferred_call["is_property"]:
                    next_qs = next_qs(*deferred_call["args"], **deferred_call["kwargs"])
                qs = next_qs
        return qs

Please let me know in the comments if you have feedback, suggestions, bugs, grave warnings, etc.

It started off pretty easy — just use “an” when the first letter of the phrase is a vowel and otherwise use “a.” And then I remembered about abbreviations (“an HR department”), silent H words (“an honorable man”), and exceptions among a few O and U words (“a one-time offer” or “a union representative”), and suddenly it wasn’t so easy.

Fortunately I managed to find an excellent database of English words that includes pronunciations, and in no time at all I was able to generate a list of exceptions.

Here’s the final result, implemented as a custom django template tag (in my use case, I needed to able to wrap the original phrase in an HTML tag, so I couldn’t use a filter tag). This could also easily be a standalone Python function as well (literally by removing the @register.simple_tag decorator).

import re

@register.simple_tag
def with_indefinite_article(phrase, before_phrase="", after_phrase=""):
    silent_h_words = ["homage", "hour", "herb", "heir", "honest", "honor"]
    letters_with_initial_vowel_sound = ["a", "e", "f", "h", "i", "l", "m", "n", "o", "r", "s", "x"]
    vowels = ["a", "e", "i", "o", "u"]
    o_exceptions = ["once","one","oneness","ones","oneself","onetime"]
    u_exceptions = ["ubiquitous","uganda","ugandan","ukraine","ukrainian","ukulele","ukuleles","unanimous","unanimously","unicellular","unicorn","unicorns","unicycle","unicycles","unification","unified","uniform","uniformed","uniformly","uniforms","unify","unifying","unilateral","unilateralism","unilaterally","union","unionist","unionists","unionization","unionized","unionizing","unions","unique","uniquely","uniqueness","unisex","unison","unisons","unit","unitarian","unitary","unite","united","uniting","units","unity","universal","universally","universe","universes","universities","university","universitys","unix","uranium","urinalysis","urinary","urinate","urinating","urine","urologist","urologists","urology","uruguay","uruguayan","uruguays","usable","usage","usages","use","used","useful","usefully","usefulness","useless","usenet","user","users","uses","usual","usually","usurp","usurpation","usurped","usurper","usurping","usury","utah","utahn","utahns","utahs","utensil","utensils","uterine","utero","uterus","utilitarian","utilities","utility","utilitys","utilization","utilize","utilized","utilizes","utilizing","utopia","utopian","utopians","utopias"]

    # get rid of everything but letters and numbers and a few characters
    first_word = re.sub(r"[^A-Za-z0-9\-]", "", phrase.split()[0])
    first_letter = first_word[0].lower()

    # default to 'a'
    article = "a"

    if first_word.isupper() or (len(first_word) == 1):
        # if all caps or a single letter, we're going to assume it's an abbreviation
        if first_letter in letters_with_initial_vowel_sound:
            article = "an"
    elif first_letter == "h" and any( [re.match(r'(?i)%s' % w, first_word) for w in silent_h_words] ):
        article = "an"
    elif first_letter in vowels:
        if first_letter == 'e':
            article = "a" if first_word.lower().split('-')[0] == 'ewe' else "an"
        elif first_letter == 'o':
            article = "a" if first_word.lower().split('-')[0] in o_exceptions else "an"
        elif first_letter == 'u':
            article = "a" if first_word.lower().split('-')[0] in u_exceptions else "an"
        else:
            article = "an"

    return "%s %s%s%s" % (article, before_phrase, phrase, after_phrase)

Hopefully this will save someone who stumbles upon this some extra work down the road. If you find any bugs, please let me know in the comments.

In today’s example, the goal was pretty straightforward. We have a model class, Topic, that has a one-to-many relationship to a ThreadedComment model class. Each ThreadedComment for each Topic, has a User associated with it; furthermore the Topic itself as a User associated with it (the creator of the topic).

The issue here is that it’s not actually a straightforward relationship between Topic and ThreadedComment, due to some complicating factors beyond the scope of this discussion. Suffice it to say what we’re looking for is a list of users who are participants in the discussion, inclusive of the topic creator. But we’re not looking for just any old list — what we need is a QuerySet, for the reason that we are making use of some specialized tags in our templates that require data to be in a QuerySet.

Here’s a quick example of what we want the interface to look like in the template:

	<ul>
	{% for participant in topic.participants %}
		<li>{{ participant.name }}</li>
	{% endfor %}
	</ul>

So let’s do this step by step.

First, let’s get a list of the ThreadedComment objects we need:

ThreadedComment.public.all_for_object(topic))

The all_for_object() method is one of the “complications” we were referring to above. All we care about is that it returns a set of ThreadedComment objects for a given topic.

Okay, but now that we have this set of objects, how are we going to return a QuerySet that yields us User objects?

My co-worker actually showed this to me, so I’ll share it here. We’ll use in keyword argument in filter(), which maps to the SQL IN clause. The Django documentation has a simple example:

Entry.objects.filter(id__in=[1, 3, 4])

is equivalent to:

SELECT ... WHERE id IN (1, 3, 4)

So that’s just what we’re going to do: pass a list of user ids as an in keyword argument to identify the users we want.

First, let’s extend our original QuerySet to get the values we need as a list:

ThreadedComment.public.all_for_object(self).values_list('user_id', flat=True)

One thing is missing — we need to include the user id of the topic creator, as I mentioned above. We’ll just evaluate the QuerySet to a list and then add the topic creator to it:

list(ThreadedComment.public.all_for_object(self).values_list('user_id', flat=True)) + [self.creator.id]

Now let’s take the list of ids and pass it to a QuerySet that we’ll return User objects. Our list goes in the pk__in keyword argument:

User.objects.filter(
    pk__in=list(
        list(ThreadedComment.public.all_for_object(self).values_list('user_id', flat=True)) + [self.creator.id]
    )
)

Now let’s add this as a method participants() to the Topic model and throw a @property decorator on it so we can call it as an attribute:

@property
def participants(self):
    return User.objects.filter(
        pk__in=list(
            list(ThreadedComment.public.all_for_object(self).values_list('user_id', flat=True)) + [self.creator.id]
        )
    )

And that’s all there is to it.

What I love about this approach is that it’s a pattern that fits in to so many places (so many that I almost feel like it deserves a higher level abstraction within the QuerySet API). There are variations that make it slightly more efficient as well; for example, if I didn’t need to append the topic creator id in my example, then I could have passed the list of ids as a query as opposed to evaluating it:

User.objects.filter(
    pk__in=ThreadedComment.public.all_for_object(self).values_list('user_id', flat=True)).query
)

The advantage here is that this is evaluated as a subselect statement, so it only requires a single database call.

Anyhow, this technique is really fantastic when you have a set of related objects that for whatever reason you aren’t able to easily obtain as a QuerySet. Enjoy.