Problems with Django Testing and Selenium

As mentioned in previous posts, I am thrilled with the Django testing infrastructure. Although TestCases are a great start, I have a situation where I really would like to automatically push some buttons on my page. This seemed like a job for Selenium. The Django docs give a nice example, so I thought I would give it a try. Things did not go as expected.

This was related to the way Django (version 1.4) runs tests and the way the staticfiles app works. When tests are run, DEBUG is set by the test runner to False. No problem when running TestCases because the pages are not actually loaded. No static files are needed.

Setup a test using LiveServerTestCase, and now static files are needed. But when DEBUG=False, the staticfiles app leaves serving static files to you. If you have never run DEBUG=False in your development environment, you probably are not setup to serve the static files. Hence you get a flood of HTTP 500 errors.

The solution is on stackoverflow. Here is my code (IS_DEV is a setting indicating the code is running on the development server):

if settings.IS_DEV and ('test' in sys.argv):
    urlpatterns += patterns('',
                            url(r'^%s/(?P.*)$'%settings.STATIC_URL[1:], 'django.views.static.serve', {
                                'document_root': settings.STATIC_ROOT,
                                }),
                            )

urlpatterns += staticfiles_urlpatterns()

When I first implemented this solution, it appeared to not be working because I still got one HTTP 500 error. This was a little difficult to debug because instead of displaying a debug error page, it just generated a server error (because DEBUG=False). I was able to track down the source of that error by turning on the logger. This showed the error was due to a missing favicon.ico file.

Django Testing Examples

Django's testing functionality is awesome. But the docs are somewhat lacking. Here's a great post on stackoverflow that will get you started. Here are a few more examples:

Fixtures

These are databases that are used only for testing. They have known, fixed content that the tests can test against. The easiest way to make a fixture is to run:

python manage.py dumpdata

The problem with that approach is it will include stuff that will break the fixture. What you need are some excludes. And to make things more readable, you might want to add some indents. Here is the command I used:

python manage.py dumpdata --indent 4 -e contenttypes -e auth.Permission -e sessions -e admin.logentry > simple_auth_fixture.json

Note, this will not remove those tables from your testing database. It will just not provide initial data for those tables.

Also, I use south for database migrations. South makes database entries and .py files to track migrations. By not excluding south there was the potential for a conflict with my migration .py files. This did not happen in the tests I did.

Testing an Object Edit View

The usual flow is for the user to load the form using initial data, then edit the data and post it. Here is one way to do that:

response = self.client.get('/app/edit_obj/3/', follow=True)
# Make sure the form loads
self.assertTemplateUsed(response, 'app/edit_obj.html')
# Get the initial form data 
form_data = response.context['form'].initial 

# Now you can change some data and post it
form_data['afield']='xyz'
response = self.client.post('/app/edit_obj/3/', form_data, follow=True)
# Use asserts to handle whatever responses you expect.

Figuring Out Unexpected Errors

Lets say you are testing a view that includes a form. You post some data and get an unexpected error. How do you figure out what the error is? Here's one way:

response = self.client.post('/app/edit_obj/3/', form_data, follow=True)

# These print statements should help you figure out what went wrong 
print 'Errors:', response.context['form'].errors
print 'Non Field Errors:',response.context['form'].non_field_errors
print 'Is Valid',response.context['form'].is_valid()

A Confusing Error from assertFormError

Here is a somewhat confusing error I got when using assertFormError. The error was:

Failure
Traceback (most recent call last):
  File "/home/junk/.virtualenvs/qdb_django14/lib/python2.7/site-packages/my_proj/an_app/tests.py", line 52, in test_call_view_fails_dup
    self.assertFormError(response, 'form', 'email', u"Email address already exists.")
  File "/home/junk/.virtualenvs/qdb_django14/lib/python2.7/site-packages/django/test/testcases.py", line 718, in assertFormError
    " response" % form)
AssertionError: The form 'form' was not used to render the response

The problem was the data I was sending to the form was supposed to be invalid. But there was an error in my form validation code. The method form.is_valid() returned true which lead to an unexpected redirect. The new page did not use the context variable 'form'. It would have been really confusing if the new page also had a context variable named 'form'.

One way to prevent this is to use assertTemplateUsed to make sure you have the correct template before you look for errors.

Testing by User Type

Often it is important to restrict certain activities to certain types of users. Since I made my fixture from a working website, it already contains the various types of users I might want to run a test on. The problem with this is I did not want to put passwords in my tests.

One way to handle that is just to reset each user's password before login. I made a method for that:

class TestEditClient(TestCase):
    def login(self,username):
        # Keep real password out of tests. Sets password to test.
        self.password='test'
        self.user = User.objects.get(username__exact=username)
        self.user.set_password(self.password)
        self.user.save()        
        response = self.client.login(username=username, password=self.password)
        if not response:
            raise RuntimeError('Could not login') 

An alternative would have been to create each user type using the standard commands for creating a user. For my project, users had a bunch of other records associated with them. I did not feel like creating those, so I just used the users we already had in the db.

Speeding Up Django on Webfaction

My Django/mod_wsgi site uses virtualenv and uses the apache directive VirtualHost. Thus the standard install files, such as httpd.conf, provided by Webfaction need modification. 

The problem I encountered was the first page hit to the server would take more than 10 seconds to load. Subsequent pages would load quickly. The Chrome auditing functionality, showed that these long load times mostly involve the browser waiting for the server after the initial http request.

Others have seen this phenomena. For me, the most useful part of this forum post was introduction to the ab tool. This tool helped me improve my httpd.conf file.

Some strangeness. The ab function continues to work and have good performance even after the server is stopped!

Chrome strangeness - I am using Version 24.0.1312.56 Ubuntu 12.04 (24.0.1312.56-0ubuntu0.12.04.1) on Linux and Version 25.0.1364.152 on Mac OS 10.7.5  I am using the developer tools on both. The network tab shows a timeline of the loading of each component of the page. I expect a bunch of my static files to be retrieved from the browser cache. Both browsers show an http status code of 200 for items that should be retrieved from cache. On the Mac the load times of those files is 0, and the page loads fast. On Linux, the page loads much slower and the load times for those components are on the order of seconds. Firefox on Ubuntu loads extremely fast and indicates it is getting those files from cache.

Caching HTTPS Static Files in Django

This post assumes you are familiar with the django staticfiles app and are already serving your static files from a different server that's optimized for serving static files.

Caching static files in the browser is an important technique for speeding up page load times. Part of the challenge in getting this right is that many of the parts are dependent on each other.

For my needs, the webpage needed to be served using HTTPS. To avoid the dreaded "mixed content" warning messsage, this meant that all of the supporting static files would also need to be served using HTTPS.

I serve my static files from a shared server on webfaction. My choices are an nginx server where my options as to set cache expires to the max or not at all. The other choice is an Apache server with htaccess controls.

The first question was, what cache controls are needed to cache HTTPS files? Some authors claim it is necessary to set cache-control:public to get the browser to cache HTTPS. If this were the case, then I would need to go with the Apache server. Evidently, that is an old recommendation. According to stackoverflow, its no longer needed as of 2010. Maybe I can still use nginx.

If I am going to use nginx, then the problem is how to update static content when the browser cache is really long. Luckily some clever Django app developers have this all figured out. They have made it possible to have collectstatic put a hash of the file content in to the filename. Thus every time you change the content, a new file is created which will cause the browser to download it and cache it.

But wait, won't changing the filename mess up my templates? Not to worry, the app the adds the hash to the filename, keeps a mapping between the original name and the new name. When a template is  rendered, the app changes the names to the new names. For more info, check out class storage.CachedStaticFilesStorage .

Django Collectstatic Gotchas

Overall the Django staticfiles app is great. But I ran into one behavior I did not expect that caused some problems.

As noted in the documentation:

 The default is to look in all locations defined in STATICFILES_DIRS and in the 'static' directory of apps specified by the INSTALLED_APPS setting

and:

Duplicate file names are by default resolved in a similar way to how template resolution works: the file that is first found in one of the specified locations will be used.

To make my apps more modular, I put a static directory in each. Each containing among other things a css folder. Unfortunately for me, I used the same name for some css files in different apps. Thus the first one found made it into collectedstatic and the other one did not. This was not what I was expecting.

To solve the problem, I set

STATICFILES_FINDERS = ( "django.contrib.staticfiles.finders.FileSystemFinder", )

And I added my apps to STATICFILES_DIRS like this:

STATICFILES_DIRS=[('my_django',os.path.join(MY_DJANGO_PATH,'static')),
('app1',os.path.join(MY_DJANGO_PATH,'app1','static')),
('app2',os.path.join(MY_DJANGO_PATH,'app2','static')))

Now collected static made a folder for each app.

Problems Stopping Matplotlib FuncAnimation

I am writing a variant of the matplotlib strip chart example. I am running matplotlib version 1.1 on Ubuntu 12.04 from Aptana Studio. I stop the animation using the close button on the plot window.

When I do that, I get the following error message:

invalid command name "182058716callit"
    while executing
"182058716callit"
    ("after" script)
Traceback (most recent call last):
  File "/home/xxxxx/Documents/Aptana Studio 3 Workspace/foot_force/plot_boxing.py", line 179, in 
    plt.show()
  File "/usr/lib/pymodules/python2.7/matplotlib/pyplot.py", line 139, in show
    _show(*args, **kw)
  File "/usr/lib/pymodules/python2.7/matplotlib/backend_bases.py", line 109, in __call__
    self.mainloop()
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_tkagg.py", line 69, in mainloop
    Tk.mainloop()
  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 328, in mainloop
    _default_root.tk.mainloop(n)
AttributeError: 'NoneType' object has no attribute 'tk'

Its not the end of the world. The code still runs as needed. But the error message is ugly, especially if the code is being used to demo something to a customer.

Turns out the solution is simple. FuncAnimation has a keyword call interval, which is the time between calls to your update function in milliseconds. Increase the interval and the problem goes away.

Using Python Logging Across Multiple Modules

For some reason, I do not use the builtin logging module in Python as often as I think I should. I am not sure why. I have used logging several times before, yet it has not become part of my standard tool kit. My first suspicion is that I am just being lazy and print statements are so easy. But now as I wade back into the waters, I am starting to remember that logging uses some form of magic that I never took the time to get comfortable with.

The magic appears when you use logging across multiple modules. Suppose you have an app called app.py that loads a function from a module as in the following code:

# app.py
from my_module import my_func
my_func()

# my_module.py
import logging
logger = logging.getLogger(__name__)

def my_func():
    logger.error('message from mod 1')
    logger.debug('a debugging message')

If you run app.py you will get the error:

No handlers could be found for logger "my_module"

But add an import in app.py and suddenly the error in my_module.py goes away:

# app.py
from my_module import my_func
import logging

logging.error('A message from app.py')
my_func()

I lied, you also have to have line 5.

I cannot think of another case where an import in one module effects another module. That's why I did not like using logger. I never got comfortable with how it worked across modules. But magic is also powerful. Time to learn it. Before we go any further, you should read the basic tutorial in the Python docs.

The questions we are going to address are:

• how does logging come to life?
• how does it stay alive when execution passes to a new module?

The answer is that once the logger is configured, it becomes part of the Python interpreter process that is executing your code. It effectively becomes global.

Part of the confusion also comes from how easy it is to configure logging. As shown in the snippet above, simply importing logging and logging a message auto-magically configures a logger for your entire project. Convenient, but un-nerving if you do not know its happening.

Here is a simple example of a more explicitly configured logger:

# app.py
from my_module import my_func
import logging
logging.basicConfig(filename='example.log',level=logging.DEBUG)

logging.error('A message from app.py')
my_func()

The logger in the previous example wrote to the console. This logger writes to a file and changes the default level. The amazing thing here is that the log messages in mod1.py also show up in this file.

This also explains why the line:

logger = logging.getLogger(__name__)

shows up in so many modules. This line causes the logging messages to be prepended with the module name. Pretty helpful when you have 47 different modules writing to the same log file.

I think that does it for me. With these mysteries cleared up, the rest of the Python docs are pretty straight forward.

But since we have gotten this far, I want to share a debugging trick. Often when I am debugging a module, I run it as __main__ along with some setup code. Here is how to make the logger work with __main__ and when it is called as part of a larger app:

# mod1.py
import logging
logger = logging.getLogger(__name__)

def mod1():
    logger.error('message from mod 1')
    logger.debug('a debugging message')
    
if __name__=='__main__':
    logging.basicConfig(level=logging.DEBUG)
    mod1() 

This sets up a different logger than the one when the module is not __main__. In the app, you may have the level set at ERROR. But when running the module as main, you can have the level be DEBUG. Also, this debugger writes to the console, which is much more convenient than writing to a file. It essentially creates print statements that can be turned on and off.

Evernote and Alternatives

I just started using Evernote. I like it because it is cloud based, so in theory my notes will be available from anywhere for several years. It also has a nice browser plugin for taking shots of webpages. All of this stuff is findable using the EN search function. I am told EN even does OCR on images and scans pdfs so that all that content is findable using the EN search function.

The one thing I do not like about EN is it's editor for creating notes within EN. To formatting is critical and the EN editor is inadequate.

One way around that prob is to use Google Docs. But the procedure for linking/synching EN w GD is almost non-existent.

I almost never create extensive, formated docs w my phone. So I tried using LibreOffice (Open Office) to create notes in HTML and then upload them to EN. Uploading worked fine, but it appears that the content did not get scanned by EN, because it was not findable with the EN search function. Evidently (as of April 2012) the only text-file-like format EN scans is PDF.

I tried opening the html doc in Chrome and using the EN snapshot pluggin. It failed.

Python Video Player Part Duex

After too many hours I gave up on MLT. Their documentation is pretty bad. I could not get "multi" working. Back to gstreamer.

Initially I was reluctant to use the gstreamer command line; get-launch. But I was not making headway in gstreamer in python, so I thought it would be good to get python out of the way for now.

My ultimate goal is to figure out how to use "tee". But I could not find good documentation on the, so switched to getting mux to work, since it has one input and 2 outputs, like tee.

I still could not get things to work. I could run the gst-launcher but it would PAUSE and I could not figure out how to get it to START. It seemed like it had something to do with putting queues on the mux output. Then all was revealed in this post. For the mpegdemux, the queues need to go after the decoders... frustrating. Here is code that works (just put your path to your mpg file):

#!/bin/bash
gst-launch-0.10 -v --gst-debug-level=1 \
mpegdemux name=demuxer \
demuxer. ! mpeg2dec ! queue ! ffmpegcolorspace ! xvimagesink \
demuxer. ! mad ! queue ! audioconvert ! pulsesink \
filesrc location="PATH TO .mpg FILE" ! demuxer.

Now to tee the video. OMG - this worked! Two identical video windows (note: they are positioned on top of each other).

#!/bin/bash
gst-launch-0.10 -v --gst-debug-level=1 \
mpegdemux name=demuxer \
demuxer. ! mpeg2dec ! queue ! ffmpegcolorspace ! tee name=my_tee \
my_tee. ! queue ! xvimagesink \
my_tee. ! queue ! xvimagesink \
demuxer. ! mad ! queue ! audioconvert ! pulsesink \
filesrc location="/home/chuck/Desktop/test.mpg" ! demuxer.

Life is good. Here it's working in python:

import sys, os
import pygtk, gtk, gobject
import pygst
pygst.require("0.10")
import gst

FILEPATH='MY FILE PATH.mpg'

class GTK_Main:
   def __init__(self):
       window = gtk.Window(gtk.WINDOW_TOPLEVEL)
       window.set_title("Mpeg2-Player")
       #window.set_default_size(200, 100)
       window.connect("destroy", gtk.main_quit, "WM destroy")
       vbox = gtk.VBox()
       window.add(vbox)
       hbox = gtk.HBox()
       vbox.pack_start(hbox, False)
       self.button = gtk.Button("Start")
       hbox.pack_start(self.button, False)
       self.button.connect("clicked", self.start_stop)
       window.show_all()
      
       self.player = gst.Pipeline("player")
       source = gst.element_factory_make("filesrc", "file-source")
       source.set_property("location", FILEPATH)
      
       demuxer = gst.element_factory_make("mpegdemux", "demuxer")
       demuxer.connect("pad-added", self.demuxer_callback) # evidently demux needs to be setup dynamically
      
       self.video_decoder = gst.element_factory_make("mpeg2dec", "video-decoder")
       self.colorspace = gst.element_factory_make("ffmpegcolorspace", "colorspace")
       self.audio_decoder = gst.element_factory_make("mad", "audio-decoder")
       audioconv = gst.element_factory_make("audioconvert", "converter")
       audiosink = gst.element_factory_make("pulsesink", "audio-output")
       videosink1 = gst.element_factory_make("xvimagesink", "video-output1")
       videosink1.set_property("name", "vid1")
       videosink2 = gst.element_factory_make("xvimagesink", "video-output2")
       videosink2.set_property("name", "vid2")
       self.tee=gst.element_factory_make("tee","my_tee")
      
       self.queuea = gst.element_factory_make("queue", "queuea")
       self.queuev = gst.element_factory_make("queue", "queuev")
       self.queuet1 = gst.element_factory_make("queue", "queuet")
       self.queuet2 = gst.element_factory_make("queue", "queuet2")
      
       self.player.add(source, demuxer, self.video_decoder, self.audio_decoder, audioconv,
           audiosink, videosink1, videosink2, self.queuea, self.queuev,self.queuet1,
           self.queuet2, self.colorspace,self.tee)
       gst.element_link_many(source, demuxer)
       gst.element_link_many(self.video_decoder,self.queuev, self.colorspace, self.tee )
       gst.element_link_many(self.audio_decoder,self.queuea, audioconv, audiosink)
       gst.element_link_many(self.tee, self.queuet1, videosink1)
       gst.element_link_many(self.tee, self.queuet2, videosink2)
      
       bus = self.player.get_bus()
       bus.add_signal_watch()
       bus.enable_sync_message_emission()
       bus.connect("message", self.on_message)
      
   def start_stop(self, w):
       if self.button.get_label() == "Start":
           self.button.set_label("Stop")
           self.player.set_state(gst.STATE_PLAYING) # tells mux it needs to connect to something, then starts
       else:
           self.player.set_state(gst.STATE_NULL)
           self.button.set_label("Start")
                      
   def on_message(self, bus, message):
       t = message.type
       if t == gst.MESSAGE_EOS:
           self.player.set_state(gst.STATE_NULL)
           self.button.set_label("Start")
       elif t == gst.MESSAGE_ERROR:
           err, debug = message.parse_error()
           print "Error: %s" % err, debug
           self.player.set_state(gst.STATE_NULL)
           self.button.set_label("Start")
  
   def demuxer_callback(self, demuxer, pad):
       if pad.get_property("template").name_template == "video_%02d":
           qv_pad = self.video_decoder.get_pad("sink")
           pad.link(qv_pad)
       elif pad.get_property("template").name_template == "audio_%02d":
           qa_pad = self.audio_decoder.get_pad("sink")
           pad.link(qa_pad)
      
GTK_Main()
gtk.gdk.threads_init()
gtk.main()

Python Video Player

For a research project, I am trying to create a python based video player that will allow me to create multiple windows that play a single video, all synchronized.

I started out with gstreamer. It looked promising. It even had a element called "Tee" which seemed like a good thing for splitting the source video. Alas, the documentation was lacking. Especially regarding "Tee".

It seems I am not the first person to run into this problem. This article made me decide to switch to MLT. Openshot seems like a pretty developed, live project. If this guy was willing to put this much work into an MLT based project, then I thought I would give it a try.

Unfortunately things did not go much better in MLT. First, I had trouble installing it (Ubuntu 11.10). I solved that problem by installing Openshot. I installed the python bindings using the Ubuntu Software Center.

Next I found the docs to be pretty cryptic - given that I wanted to work in python. Eventually I found the mlt-0.7.8.tar.gz file. In ./src/swig/python was some demo code that was enough to get me started.

Below is some code that will open a media file and play it. Just assign the full path FILE and run.

import mlt
import os.path
import time
import sys

FILE='/home/chuck/Desktop/test-mpeg.mpg'
if os.path.exists(FILE):
# Start the mlt system
mlt.Factory().init( )

# Establish a profile
profile = mlt.Profile( )

# Create the producer
p = mlt.Producer( profile, FILE  )

# Create the consumer
c = mlt.Consumer( profile, "sdl" )

# Turn off the default rescaling
c.set( "rescale", "none" )

# Connect the producer to the consumer
c.connect( p )

# Start the consumer
c.start( )

# Wait until the user stops the consumer - ie closes the window
while c.is_stopped( ) == 0:
    print 'look, we can do other stuff while the video runs'
    time.sleep( 1 )
print 'The window was closed.'
else:
# Diagnostics
print "Unable to open ", FILE

Not sure how I am going to solve my original problem. But this is a start.

Running Django Celery as a Background Process on Webfaction w Virtualenv

I am using Django Celery as a scheduler. I also use virtualenv. I got it to the point were it would work if I ran:

workon my_env

python manage.py celeryd -v 2 -B -s celery -E -l INFO

but when I closed the SSH window, this process stopped. The challenge was to figure out how to keep the process running.

Most of the information I needed was in: http://ask.github.com/celery/cookbook/daemonizing.html#init-script-celerybeat . Thing I was not sure about was what to do about the defaults paths:

/etc/init.d

/etc/default

since I do not think I can put stuff in those paths on webfaction. What I did instead is put a celery dir in my root. In that dir I put the dirs init.d and defult. In ~/celery/init.d I put the script celeryd that I got from https://raw.github.com/ask/celery/master/contrib/generic-init.d/celeryd

If you look at that script, there are lots of things it would seem that should be changed. However most of those changes can be made in the config file. A sample of the config file is at:

http://ask.github.com/celery/cookbook/daemonizing.html#example-django-configuration

chmod 770

Edit ~/celery/init.d/celeryd to change the value of CELERY_DEFAULTS to point to our config file in ~/celery/default/celeryd

I got the values for CELERYD_USER and CELERYD_GROUP by running the command:

id

I created 2 more dirs in ~/celery: logs and run.

Facebook and Pinax Avatars

Pinax is bundled with an app for managing avatars. If your users are uploading their photos, then using the avatar app is pretty straight forward. If you are grabbing a avatar from a users Facebook site, then things are not as easy.

The photo can be accessed using:

file = urllib.urlopen("https://graph.facebook.com/%s/picture?type=normal"%(user['id'],))

The problem comes when you try to save the photo to an avatar using something like this:

avatar=Avatar(user=request.user)

new_file = avatar.avatar.save(path, file.read())

avatar.save()

I was able to get it to work by saving it to a temp file, then wrapping it in a django File:

from tempfile import NamedTemporaryFile

from django.core.files import File

from avatar.models import Avatar,avatar_file_path

file = urllib.urlopen("https://graph.facebook.com/%s/picture?type=normal"%(user['id'],)) fp=NamedTemporaryFile(delete=True)

fp.write(file.read())

avatar=Avatar(user=request.user)

path = avatar_file_path(user=request.user,filename='facebook_%s.jpg'%date.today())

avatar.avatar.save(os.path.join(settings.MEDIA_ROOT,path),File(fp))

avatar.save()

fp.close()

I tried using StringIO, but could not get it to work.

Django Social-registration and the Facebook API

I am creating a Pinax based website that will require users to have an account and to login. I wanted users with Facebook accounts to be able to use OAuth and Facebook to create their account on my site. I also wanted to use data from Facebook to fill in as much of the user profile as possible. To accomplish this, I installed the app social-registration.

Facebook provides basic info such as first name and last name without extended permissions. Email address requires extended access. Extended access can be requested using the scope parameter during facebook login.

Social-registration has some javascript (templates/socialregistration/facebook_js.html ) that you can put in your templates to allow users to login with facebook. In that js file, the code that adds the scope parameter is commented out:

FB.login(handleResponse/*,{perms:'publish_stream,sms,offline_access,email,read_stream,status_update,etc'}*/);

Uncommenting that code solves part of the problem. However, the perms list is really just a placeholder. status_update and etc are not valid perms, but the rest are. Remove those and it should work.

If you put the social-registration middleware in your settings, then you can access all the users facebook info in your views using:

data_dict=request.facebook.graph.get_object('me')

Pinax Static Media

Pinax 0.7 uses the django app called staticfiles to manage...well... static files. This app is installed in the virtualenv site-packages directory.

The thing that makes this app a little hard to understand at first is it's not very DRY. The way it works is you scatter your static files all over your django project (with rules about where each dir should be). In your settings.py file you create a variable called STATICFILES_DIRS. It looks something like this:

STATICFILES_DIRS = (

('pinax', os.path.join(PINAX_ROOT, 'media', PINAX_THEME)),

('skatepark', os.path.join(PROJECT_ROOT, 'media')),)

Then you run:

python manage.py build_media --all

Note this command is different in version 0.9. This command goes to each dir and copies all the subdirs and files into the dir specified by the settings.py variable STATIC_ROOT. For example:

STATIC_ROOT = os.path.join(PROJECT_ROOT, 'site_media', 'static')

I guess in the end, putting all your static files in one place makes it easier to server them on the production server. At this point in my project, the value of the staticfiles app is it makes it easy to over-ride default static media. In my case I wanted to over-ride the pinax logo.

The default was at:

/home/me/envs/skatepark/lib/python2.7/site-packages/pinax/media/default/pinax/images/logo.png

I put the logo I wanted in my project dir:

/home/me/skatepark/media/pinax/images/logo.png

The command

python manage.py build_media --all

started with the first dir in STATICFILES_DIR being copied and the default logo being written to:

/home/chuck/skatepark/site_media/static/pinax/images/logo.png

then the next dir in STATICFILES_DIR was scanned and the default logo was overwritten by my logo.

Testing Facebook API on Local Host

I am developing a django site using django-socialregistration. I am using the django development server.

I setup the facebook app via http://www.facebook.com/developers/apps.php

This page let me get the API_KEY and SECRET_KEY.

Without setting any other settings, I got this error:

API Error Code: 191
API Error Description: The specified URL is not owned by the application
Error Message: redirect_uri is not owned by the application.

The facebook app settings include settings for:

• Site URL
• Site Domain

When I set:

Site URL: http://127.0.0.1:8000/

Site Domain:

I was able to test the api.

Setting up a Django with Pinax site on Webfaction Scratch

These are my notes for setting up a Django with Pinax website on Webfaction. The code will be in a virtualenv. We will be using Python2.5, mod_wsgi.

I have been using webfaction for over 3 years. I have a few separate accounts for different projects. All of my sites are very low volume (but high value?). I have nothing but good things to say about them. There support is phenomenal. They have helped me with many things that were essential to my python projects, but far beyond their responsibilities as a server host. Their up-time is great, it must be near 99%. I could not be happier with all they have done.

To start off, I created an account on webfaction. If all we were going to do is run one Django site we probably could have gotten by with Shared Plan 1. But my kids are noobs and I want them to feel free to put stuff on this server so we went w Shared Plan 2. But the most important point is we are starting with a naked webfaction account. It took webfaction less than an hour to setup this account.

My local machine is windows XP - so I use putty to get shell access (SSH) to the account. Webfaction has a page on Accessing Your Data.

After connection with SSH, I installed virtualenv. I use Python 2.5 because many of the libraries I like to use do not work with later versions. To install virtualenv, I ran the command:

easy_install-2.5 virtualenv

To make using virtualenv easier to use I also installed virtualenvwrapper. It needed pip, so I installed that first:

easy_install-2.5 pip

pip install virtualenvwrapper

In my home directory I added a dir to hold all the virtualenvs:

mkdir ve

I edited by .bashrc file by adding the following lines:

export WORKON_HOME=$HOME/ve

VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python2.5

source /home/myaccount/bin/virtualenvwrapper.sh

Then I restarted the shell.

The goal of this website is to support our attempt to make a local indoor skateboard park. So I made a virtualenv for that:

mkvirtualenv --no-site-packages skatepark

I used the --no-site-packages flag to provide the most isolation. Click here for more info.

Next step is to put all the python packages that we will need into the virtualenv. First activate the env, then go to the site-packages dir by running the commands:

workon skatepark

cdsitepages

I installed the latest stable version of Pinax:

*******whoops - pinax installs itself as a virtualenv! ****** change in plans:

cd ~

rmvirtualenv skatepark

wget http://downloads.pinaxproject.com/Pinax-0.7.3-bundle.tar.gz

tar zxf Pinax-0.7.3-bundle.tar.gz

python2.6 Pinax-0.7.3-bundle/scripts/pinax-boot.py ~/ve/skatepark

workon skatepark

cdsitepackages

The rest of this install is loosely based on this. To install additional dependencies:

pip install http://effbot.org/downloads/Imaging-1.1.7.tar.gz

wget http://initd.org/psycopg/tarballs/psycopg2-2.2.2.tar.gz

tar zxf psycopg2-2.2.2.tar.gz

cd psycopg2-2.2.2

At this point the docs on pinax for installing on webfaction say to edit setup.cfg so that:

#pg_config=

becomes:

pg_config=/usr/local/pgsql/bin/pg_config

This lead to an error when I did the next step. I posted a question on the webfaction discussion forum and the tech guys at webfaction posted an answer almost immediately. It turns out they changed the way webfaction was configured. The new location is:

pg_config=/usr/bin/pg_config

The commands:

python setup.py build

python setup.py install

completed the installation of psycopg2.

Time to create a project:

cdsitepackages

mkdir projects

cd projects

mkdir skatepark

Next I edited settings.py - something like this:

SERVE_MEDIA = False

DATABASE_ENGINE = 'postgresql_psycopg2'

DATABASE_NAME = 'my_account_myproject'

DATABASE_USER = 'my_account_myproject'

DATABASE_PASSWORD = 'WebFaction when you created the database>'

Next I setup the cron system for sending email. The notes for this were rather vague and possibly wrong. So here is what I did (from the skatepark dir):

mkdir logs

mkdir cron

cd cron

vi send_mail.sh

Paste the following into send_mail.sh:

#!/bin/sh

WORKON_HOME=~/ve/skatepark

PROJECT_ROOT=~/ve/skatepark/lib/python2.5/site-packages/projects/skatepark/

# activate virtual environment

. $WORKON_HOME/bin/activate

cd $PROJECT_ROOT

python manage.py send_mail >> $PROJECT_ROOT/logs/cron_mail.log 2>&1

Next copy send_mail.sh to retry_deferred.sh and to emit_notices.sh. Edit these two files to change the command sent to manage.py.

Add execute permission to these files:

chmod +x send_mail.sh

chmod +x retry_deferred.sh

chmod +x emit_notices.sh

Use command "crontab -e" to add these scripts to your cron file:

* * * * * ~/ve/skatepark/lib/python2.5/site-packages/projects/skatepark/cron/send_mail.sh

* * * * * ~/ve/skatepark/lib/python2.5/site-packages/projects/skatepark/cron/emit_notices.sh

0,20,40 * * * * ~/ve/skatepark/lib/python2.5/site-packages/projects/skatepark/cron/retry_deferred.sh

Problem Migrating Multilingual to Django 1.2

I just upgraded a multilingual Django project to v1.2.

To upgrade to the new version of the multilingual app, I switch to the django-multilingual-ng branch. That branch involved some changes in the database structure.

As recommended, I installed the app south to migrate the database. Then I ran the mlng_convert command on my apps that use multilingual.

The problem I encountered was caused by apps whose models refer to other apps via foreign keys. Evidently south and mlng_convert follows those relationships and updates those table as well.

If I ran mlng_convert on a table that was already converted, I got the database error (MySQL) "Duplicate column name 'language_code'"

More Webfaction, virtualenv --no-site-packages

I found a way to get the --no-site-packages flag for virtualenv to work "good enuf" on webfaction. The solution involved reading this forum discussion more carefully. The mojo (hack) is in post #7.

My plan is to put packages that are of general use and that rarely change in ~/lib/python2.X. These will not be available in my virtualenv. For standard packages that I want in the virtualenv, I will manually add paths to them using the virtualenvwrapper command add2virtualenv.

Update - March 1, 2013
The link above no longer works. Here is a new link, with a new and improved discussion.

By the way, if you are not sure what packages are being imported, try this:

1. Activate your virtualenv 
2. Start python from the command line
3. Type import sys
4. Type sys.path

Webfaction, virtualenv and --no-site-packages

So I thought I had finally figured out how I wanted to configure my code on webfaction using env. My plan involved using the flag --no-site-packages.

Once I created the virtualenv, I fired up python and ran the code:

import sys

for x in sys.path:

print x

Much to my surprise, all "my site packages" (/username/lib/python2.5 ) paths were in the system path. I was hoping these packages would not be included.

This post on the webfaction forum explained what was happening. Basically the --no-site-packages flag removes the ones installed by webfaction for all users.

Next, I tried to setup the paths using a sitecustomize.py file. When python starts, it looks for that file. If it finds it, it imports it. Unfortunately that did not work either because webfaction already has one in /usr/local/lib/python2.5.

I tried adding an import statement to a .pth file. That solution did not work because there is no guarantee of when that file will be executed relative to the other .pth files. In my case, it ran before the paths I wanted to remove from sys.path.

For now, here's my soln. The only project I have running in virtualenv is django. I am just going to add an import statement in manage.py. It will configure the site path. On my development computer, I am not having all these problems. So I will put the import statement in a try block. If the config program is not found, it will just move on.

virtualenv and --no-site-packages

I originally created my virtualenv without the flag --no-site-packages. I figured that the virtualenv paths would over-ride (come before) the global paths. That turned out to be wrong. The global paths come first.

One way to get around this is the create the env with the --no-site-packages flag and then manually add the global packages that I want using the virtualenvwrapper command add2virtualenv.

Since the env was already setup, I looked for ways to add that flag after the repo was created. Here is the answer. To have virtualenv not use global libs, just add the file "no-global-site-packages.txt" to the path: virtualenv_name/lib/python2.5 - Note this path is different from the one given on stackoverflow. I figured out the path by creating a test virtualenv and seeing where that file turned up.

But that's not really what I want to do. I want to have my env access the global paths because there are a lot of useful libs in there. It would be a lot of work manually adding each to the virtualenv.

Part of the answer seems to be in the pth files. It turns out that when you install a package in the virtual env using easy_install, easy_install creates a .pth file that removes versions of that package that are already on the path.

A quick peak in easy-install.pth reveals lines of code in addition to the paths! Here it is:

import sys; sys.__plen = len(sys.path)

./setuptools-0.6c11-py2.5.egg

./pip-0.7.2-py2.5.egg

import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

I found more info on this magic here. Apparently if you start a line with import and put all the code after it on the same line using ";" as a delimiter, then you can hack the path at startup.