Setting up a Django environment and project structure

While getting to grips with Django for my own project, I’ve been keeping notes on exactly what I’ve done so I can repeat it in future. And spending some time working out exactly how to structure the project — in terms of both files and software used — so that I can also repeat that. I want a documented process so that I can get my forgetful self up and running with a consistent new project as quickly as possible.

This post is my record of the process, and I’ll keep it updated as I change my mind or add new things. If you have any suggestions I’d love to hear them; I’m new to Django and haven’t used most of these tools — virtualenv, mkvirtualenv, pip, git, github — a huge amount before. Some of this is very basic, but I wanted to document everything I need to do.

So far this doesn’t include getting a live server working. NOTE: This was written in 2010 and should be accurate for Django 1.2 and 1.3 but I think some of the project structure, and maybe settings, have changed with Django 1.4.

For further reading, here are some articles I found useful while working on this:


File structure

I want to keep a similar structure for all my projects. For example, this is the location and structure I have for django-hines:

phil/
    .virtualenvs/
        django-hines/
    Projects/
        personal/
            django-hines/
                .gitignore
                hines/
                    __init__.py
                    aggregator/
                        __init__.py
                        admin.py
                        managers.py
                        migrations/
                        models.py
                        templates/
                            aggregator/
                                day.html
                                index.html
                            base_aggregator.html
                        templatetags/
                            __init__.py
                            aggregator_tags.py
                        test.py
                        urls.py
                        views.py
                    context_processors.py
                    manage.py
                    settings_default.py
                    settings.py
                    settings.py.template
                    static/
                        css/
                        img/
                        js/
                    templates/
                        404.html
                        500.html
                        base.html
                        comments/
                        flatpages/
                    weblog/
                logs/
                scripts/
                README.mdown
                requirements.txt

aggregator and weblog are two apps within the hines project (I haven’t expanded the weblog directory, but it’s the same as the aggregator one). Templates specific to an app are within their own projectname/appname/templates/ directory. Common project templates, or changes to third-party templates are in the projectname/templates/ directory. Everything within django-hines is checked in to git, except for hines/settings.py which are the settings local to my environment.


Setting up the environment

To install virtualenv (which includes ):

$ easy_install virtualenv

That might install pip, but at least once I’ve found it didn’t. In which case you need to install it yourself:

$ easy_install pip

Then either install virtualenvwrapper:

$ sudo pip install virtualenvwrapper

or maybe upgrade an existing version:

$ sudo pip install --upgrade virtualenvwrapper

(The sudo might or might not be required or allowed, depending on your server.)

Then, if it doesn’t already exist:

$ mkdir ~/.virtualenvs

And put this in ~/.bashrc:

export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
export PIP_VIRTUALENV_BASE=$WORKON_HOME # Tell pip to create its virtualenvs in $WORKON_HOME.
export PIP_RESPECT_VIRTUALENV=true # Tell pip to automatically use the currently active virtualenv.

You may need to replace /usr/local/bin/ with a different path. For example, on one shared server I use /home/philgyford/bin/.

Those instructions will be loaded each subsequent time you log in. To make them take effect for this session, do this:

$ source ~/.bashrc

Then:

$ mkvirtualenv --no-site-packages --distribute django-projectname

That creates and starts you working on django-projectname. Install other things with pip:

(django-projectname)$ pip install yolk
(django-projectname)$ yolk -l
(django-projectname)$ pip install Django
(django-projectname)$ pip install MySQL-python
(django-projectname)$ pip install django-debug-toolbar
(django-projectname)$ pip install south

Instead of installing MYSQL-python you might want to install psycopg2 instead, if you’re using Postgres.

Note that we’ve installed South for managing database migrations there. More on that in a moment.

Add the directory where we’ll keep files:

(django-projectname)$ mkdir ~/Projects/subdir/django-projectname
(django-projectname)$ cd ~/Projects/subdir/django-projectname
(django-projectname)$ mkdir logs
(django-projectname)$ django-admin.py startproject projectname
(django-projectname)$ mkdir projectname/templates

Now we’ll have ~/Projects/subdir/django-projectname/projectname/ with a templates/ directory within it.

Set up a database:

$ mysql -u root -p
mysql> CREATE DATABASE projectname DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
mysql> GRANT ALL ON projectname.* TO username@localhost IDENTIFIED BY 'password';

Set the DATABASES settings in ~/Projects/subdir/django-projectname/projectname/settings.py. Then:

(django-projectname)$ ./manage.py runserver

And then you should be able to go to http://127.0.0.1:8000/. (You might need to do something like chmod u+x manage.py to get ./manage.py... to work.)


Settings

Rename settings.py to settings\_default.py.

At the beginning of settings\_default.py add this:

import os,sys
PROJECT_ROOT = os.path.abspath(os.path.dirname(__file__))

Also, further down, set the templates directory (which we created earlier):

TEMPLATE_DIRS = (
    os.path.join(PROJECT_ROOT, 'templates'),
)

If you’re going to use South (which we installed with pip, earlier) then add it to the list of INSTALLED_APPS:

INSTALLED_APPS = (
    ...
    'south',
)

Then, create a local settings file that will override defaults:

(django-projectname)$ cd ~/Projects/subdir/django-projectname/projectname
(django-projectname)$ vi settings.py

And move any settings that are server-specific from settings\_default.py to settings.py. Suggested local settings:

  • DEBUG
  • TEMPLATE_DEBUG
  • ADMINS
  • MANAGERS
  • DATABASES
  • SECRET_KEY

At the start of settings.py, add this to import the default settings:

try:
    from settings_default import *
except ImportError:
    pass

If your project is going to be checked in to public version control, then the settings.py file should not be checked in, while the settings\_default.py can be. If your version control is private you could probably do a separate settings file for each server and check them all in.

Next, create a template for settings.py which will be checked in to version control, and used by anyone checking the project out:

(django-projectname)$ cp settings.py settings.py.template

And remove any specific settings in there to make an anonymous example.

(It seems more common to do this the other way round — have settings.py as the default settings, and create a settings\_local.py for the local overrides. But, it seems that then it’s hard to do things like INSTALLED_APPS += ('debug\_toolbar',), which we do in a moment. It generated errors, possibly something to do with scope…?)


Django-debug-toolbar

We already installed it with pip.

Add this in settings.py and settings.py.template:

INTERNAL_IPS = ('127.0.0.1',)
MIDDLEWARE_CLASSES += ('debug_toolbar.middleware.DebugToolbarMiddleware',)
INSTALLED_APPS += ('debug_toolbar',)

That IP address is correct if you’re working locally.


Git

Make the ignore file:

(django-projectname)$ cd ~/Projects/subdir/django-projectname
(django-projectname)$ vi .gitignore

And in .gitignore have something like this:

# File types #
##############
*.pyc
*.swo
*.swp
*.swn

# Directories #
###############
logs/

# Specific files #
##################
projectname/settings.py

# OS generated files #
######################
.DS_Store?
ehthumbs.db
Icon?
Thumbs.db
*~

Alternatively, having the File types and OS generated files in your own ~/.gitignore file that will apply to all projects is probably better.


Adding to Github

Make a README.mdown (or .txt or whatever) in ~/Projects/subdir/django-projectname.

Create a new project on Github. Then, assuming git is already set up on your machine:

(django-projectname)$ cd ~/Projects/subdir/django-projectname
(django-projectname)$ pip freeze > requirements.txt
(django-projectname)$ git init
(django-projectname)$ git add .
(django-projectname)$ git status
(django-projectname)$ git commit -m 'Initial commit'
(django-projectname)$ git remote add origin git@github.com:username/django-projectname.git
(django-projectname)$ git push origin master

Starting work

Probably uncomment django.contrib.admin in INSTALLED\_APPS, and the three admin-related lines in urls.py.

Set up database etc:

(django-projectname)$ cd ~/Projects/subdir/django-projectname/projectname
(django-projectname)$ ./manage.py syncdb

Create initial app:

(django-projectname)$ cd ~/Projects/subdir/django-projectname/projectname
(django-projectname)$ ./manage.py startapp appname

Once you’ve added the first model(s) to your app, you’ll need to either repeat this:

(django-projectname)$ ./manage.py syncdb

or, if you’re using South, then do this:

(django-projectname)$ ./manage.py schemamigration appname --initial
(django-projectname)$ ./manage.py migrate appname

The first line creates the migration file for the initial database schema (in appname/migrations/), and the second line applies it to the database. See “Ongoing work”, below, for what you need to do if you make subsequent changes to the app’s model(s).


Tests

Tests are done on a per-app basis, rather than per-project. After setting a site up with data you might want to test (eg, dummy weblogs and entries, flatpages, comments added through the front end, etc.)

(django-projectname)$ mkdir ~/Projects/subdir/django-projectname/projectname/appname/fixtures
(django-projectname)$ cd ~/Projects/subdir/django-projectname/projectname/appname/fixtures
(django-projectname)$ ./manage.py dumpdata appname --indent > test_data.json

This will dump the data for the specified app. You can dump the entire database by omitting the appname. This file (and those from other apps, if required) can then be loaded when you run tests. appname/tests.py can be something like:

from django.test.client import Client
from django.test import TestCase

class WeblogClientTests(TestCase):

    fixtures = [
            '../fixtures/test_data.json', 
            '../../otherappname/fixtures/test_data.json',
        ]

    def test_HomePage(self):
        '''Test if the homepage renders.'''
        c = Client()
        response = c.get('/')
        self.failUnlessEqual(response.status_code, 200)

Then run all tests on an app using:

(django-projectname)$ ./manage.py test appname

I’m not sure of the best way to update test\_data.json files as you develop the site, other than manually, or by replacing them with a fresh file.


Other tips

ContextProcessors

There are probably variables that you’ll want to appear on every page. Create a context\_processors.py file, probably within an app (although it works at project level too). Within it have function(s) like this:

def date_formats(request):
    from django.conf import settings
    return {
        'date_format_long': 'l j F Y',
    }

That would, as an example, make a date\_format\_long variable available in all of your templates.

Then in settings\_default.py add this:

import django.conf.global_settings as DEFAULT_SETTINGS
TEMPLATE_CONTEXT_PROCESSORS = DEFAULT_SETTINGS.TEMPLATE_CONTEXT_PROCESSORS + (
    'appname.context_processors.date_formats',
)

with your own appname and method name.

Then be sure to use the custom RequestContext in your views. eg:

from django.template import RequestContext

def weblog_entry_detail(request, year, month, day, slug):
    ...
    return render_to_response('weblog/entry_detail.html', {
        'entry': entry,
    }, context_instance=RequestContext(request))

Error templates

Don’t forget to make error templates for 404 and 500 errors (as mentioned here). These should be in the top level of your templates, named 404.html and 500.html. Not having them isn’t a problem while working on the site with DEBUG=True set, but as soon as you switch to “live” you’ll get reports of missing templates whenever one of those errors occurs.


Ongoing work

Virtualenv

When returning in the future, start working on a virtualenv by:

$ workon django-projectname

And stop working on it with:

(django-projectname)$ deactivate

To remove a virtual environment:

$ rmvirtualenv django-projectname

South

If you’re using South, then when you make changes to your app’s model(s) you’ll need to do this to update the database:

(django-projectname)$ ./manage.py schemamigration appname --auto
(django-projectname)$ ./manage.py migrate appname

If your model change isn’t simple you may get a prompt for further action. See Advanced Changes. If change something that means existing data needs to change, see Data Migrations.

Git

Adding new files:

(django-projectname)$ git add file_or_folder_name

Committing more work:

(django-projectname)$ git commit -a

And this again whenever you want to push to Github:

(django-projectname)$ git push origin master

Pip

To upgrade something already installed with pip, and record the change:

(django-projectname)$ cd ~/Projects/subdir/django-projectname
(django-projectname)$ pip install --upgrade django-debug-toolbar
(django-projectname)$ pip freeze > requirements.txt

Updates:

  • 16 Nov 2010: Added instructions for South.
  • 27 Jun 2011: Tweaked pip/virtualenv installation.
  • 2 May 2012: A few little tweaks.

Comments

  • Great stuff! A couple of (debatable) Git nits:

    * git commit -a is gently discouraged because Git affords a more meticulous approach: 1. make a haphazard bunch of changes to your working directory, 2. stage some meaningful, self-contained subset of those changes, 3. commit the staged changes, 4. goto 2. The commit -a sledgehammer isn’t bad in itself but it does mean you miss out on a lot of what’s good about Git workflow.

    * It’s usually better to keep OS-specific exclusions (.DS_Store and friends) out of the project’s .gitignore and put them in your personal global ignore file instead. That way you don’t have to recreate them across multiple projects, and you don’t end up adding more cruft each time you collaborate with someone on a different OS. Googling for “git global ignore” explains how to do this.

    * Beware that your process for creating a new project on GitHub leaves you with a non-tracking master branch. This is fine as long as you’re not collaborating (i.e. only pushing), but having a tracking branch can be more convenient when you’re pulling changes. http://progit.org/book/ch3-5.html#tracking_branches explains the difference.

  • Thanks Tom. Git still baffles me, hence having to write down instructions I can blindly follow! Having read that bit about tracking branches I’m still a little lost. Can you suggest an alternative process I should use when starting a new project that would be better than what I have currently? Staging also means nothing to me, so I’ll have to read up that too…

  • Yes, it’s very baffling at first. It’s worth continuing to bang your head against it until all the pieces click into place, though, because the payoff is delightful.

    I wouldn’t worry too much about the tracking branch issue as long as it doesn’t bother you, which it won’t until you start collaborating. I just wanted you to be forewarned.

    Roughly the symptom is that creating a repository locally and then pushing it up to GitHub will give you a slightly different branch configuration (vis a vis tracking) than cloning a remote repository; in the latter case the local branches will automatically track the remote ones, which just means that Git automatically knows how to move local branches to match any moves that the remote branches make. Therefore the simplest workaround is to blow away your local repository once you’ve pushed it up to GitHub and then clone it back from GitHub again, at which point it’ll magically be set up correctly.

    If you want to be more clever and less destructive, http://blog.adsdevshop.com/2009/09/29/track-maste-in-git/ explains how to tweak your branch configuration directly.

  • I’m also relatively new to git, and there are still many things I don’t fully understand, but staging is great. Not only can you only commit some of your changed files, using the magic “git add —patch FILE” you can commit only /bits/ of your changed files :) So basically you can make each commit standalone and doing one thing; though I’m still very much guilty of commit messages saying “Fixes this, and this, and changes this bit”.

    Obviously an advantage of a framework like Django is its relative ease in changing, but I thought I’d mention that if in future you plan to do any geographical query stuff (following on from your day pages, for example), PostgreSQL has much better geographic handling than MySQL, meaning you can do things like “show me all my checkins within half a mile of this one” or, say, http://mapit.mysociety.org/ rather more easily.

    Lastly in my random witterings, I find the first thing I do in any Django project is make a render() function that looks something like:

    def render(request, template_name, context=None):
        if context is None: context = {}
        return render_to_response(template_name, context,
            context_instance = RequestContext(request)
        )

    which I stick in shortcuts.py at the top level usually for all the views to use. The reason for the first line of the function (not strictly needed here, but it’s the principle) is one of Python’s few horrible gotchas (perhaps its only one :) ), which I won’t immediately reveal as the feeling of elation and revulsion as you work it out is worth it.

  • Ooh, thanks for the render() idea Matthew. I’d been looking at all those render_to_reponse()s in my views and thinking they seemed annoyingly repetitive.

    I’ve mainly stuck with MySQL because it’s what I’m most familiar with and, so far, I haven’t needed anything it couldn’t do — I’m not very demanding. Maybe I should make the switch now though…

    And it sounds like I should get to grips with staging. Just when I thought I’d learned enough Git to do the few things I need!

29 Sep 2010 at Twitter