Programmer's Notes and Random Random Thoughts

Facebook and Pinax Avatars

2011-01-30T12:12:00.000-08:00

Pinax is bundled with an app for managing avatars. If your users are uploading their photos, then using the avatar app is pretty straight forward. If you are grabbing a avatar from a users Facebook site, then things are not as easy.

The photo can be accessed using:

file = urllib.urlopen("https://graph.facebook.com/%s/picture?type=normal"%(user['id'],))

The problem comes when you try to save the photo to an avatar using something like this:

avatar=Avatar(user=request.user)

new_file = avatar.avatar.save(path, file.read())

avatar.save()

I was able to get it to work by saving it to a temp file, then wrapping it in a django File:

from tempfile import NamedTemporaryFile

from django.core.files import File

from avatar.models import Avatar,avatar_file_path

file = urllib.urlopen("https://graph.facebook.com/%s/picture?type=normal"%(user['id'],)) fp=NamedTemporaryFile(delete=True)

fp.write(file.read())

avatar=Avatar(user=request.user)

path = avatar_file_path(user=request.user,filename='facebook_%s.jpg'%date.today())

avatar.avatar.save(os.path.join(settings.MEDIA_ROOT,path),File(fp))

avatar.save()

fp.close()

I tried using StringIO, but could not get it to work.

Django Social-registration and the Facebook API

2011-01-23T13:09:00.000-08:00

I am creating a Pinax based website that will require users to have an account and to login. I wanted users with Facebook accounts to be able to use OAuth and Facebook to create their account on my site. I also wanted to use data from Facebook to fill in as much of the user profile as possible. To accomplish this, I installed the app social-registration.

Facebook provides basic info such as first name and last name without extended permissions. Email address requires extended access. Extended access can be requested using the scope parameter during facebook login.

Social-registration has some javascript (templates/socialregistration/facebook_js.html ) that you can put in your templates to allow users to login with facebook. In that js file, the code that adds the scope parameter is commented out:

FB.login(handleResponse/*,{perms:'publish_stream,sms,offline_access,email,read_stream,status_update,etc'}*/);

Uncommenting that code solves part of the problem. However, the perms list is really just a placeholder. status_update and etc are not valid perms, but the rest are. Remove those and it should work.

If you put the social-registration middleware in your settings, then you can access all the users facebook info in your views using:

data_dict=request.facebook.graph.get_object('me')

Pinax Static Media

2011-01-06T16:07:00.000-08:00

Pinax 0.7 uses the django app called staticfiles to manage...well... static files. This app is installed in the virtualenv site-packages directory.

The thing that makes this app a little hard to understand at first is it's not very DRY. The way it works is you scatter your static files all over your django project (with rules about where each dir should be). In your settings.py file you create a variable called STATICFILES_DIRS. It looks something like this:

STATICFILES_DIRS = (

('pinax', os.path.join(PINAX_ROOT, 'media', PINAX_THEME)),

('skatepark', os.path.join(PROJECT_ROOT, 'media')),)

Then you run:

python manage.py build_media --all

Note this command is different in version 0.9. This command goes to each dir and copies all the subdirs and files into the dir specified by the settings.py variable STATIC_ROOT. For example:

STATIC_ROOT = os.path.join(PROJECT_ROOT, 'site_media', 'static')

I guess in the end, putting all your static files in one place makes it easier to server them on the production server. At this point in my project, the value of the staticfiles app is it makes it easy to over-ride default static media. In my case I wanted to over-ride the pinax logo.

The default was at:

/home/me/envs/skatepark/lib/python2.7/site-packages/pinax/media/default/pinax/images/logo.png

I put the logo I wanted in my project dir:

/home/me/skatepark/media/pinax/images/logo.png

The command

python manage.py build_media --all

started with the first dir in STATICFILES_DIR being copied and the default logo being written to:

/home/chuck/skatepark/site_media/static/pinax/images/logo.png

then the next dir in STATICFILES_DIR was scanned and the default logo was overwritten by my logo.

Testing Facebook API on Local Host

2011-01-02T07:21:00.000-08:00

I am developing a django site using django-socialregistration. I am using the django development server.

I setup the facebook app via http://www.facebook.com/developers/apps.php

This page let me get the API_KEY and SECRET_KEY.

Without setting any other settings, I got this error:

API Error Code: 191
API Error Description: The specified URL is not owned by the application
Error Message: redirect_uri is not owned by the application.

The facebook app settings include settings for:

Site URL
Site Domain

When I set:

Site URL: http://127.0.0.1:8000/

Site Domain:

I was able to test the api.

Problem Migrating Multilingual to Django 1.2

2010-11-11T09:39:00.000-08:00

I just upgraded a multilingual Django project to v1.2.

To upgrade to the new version of the multilingual app, I switch to the django-multilingual-ng branch. That branch involved some changes in the database structure.

As recommended, I installed the app south to migrate the database. Then I ran the mlng_convert command on my apps that use multilingual.

The problem I encountered was caused by apps whose models refer to other apps via foreign keys. Evidently south and mlng_convert follows those relationships and updates those table as well.

If I ran mlng_convert on a table that was already converted, I got the database error (MySQL) "Duplicate column name 'language_code'"

More Webfaction, virtualenv --no-site-packages

2010-09-02T07:26:00.000-07:00

I found a way to get the --no-site-packages flag for virtualenv to work "good enuf" on webfaction. The solution involved reading this forum discussion more carefully. The mojo (hack) is in post #7.

My plan is to put packages that are of general use and that rarely change in ~/lib/python2.X. These will not be available in my virtualenv. For standard packages that I want in the virtualenv, I will manually add paths to them using the virtualenvwrapper command add2virtualenv.

Webfaction, virtualenv and --no-site-packages

2010-08-29T14:00:00.000-07:00

So I thought I had finally figured out how I wanted to configure my code on webfaction using env. My plan involved using the flag --no-site-packages.

Once I created the virtualenv, I fired up python and ran the code:

import sys

for x in sys.path:

print x

Much to my surprise, all "my site packages" (/username/lib/python2.5 ) paths were in the system path. I was hoping these packages would not be included.

This post on the webfaction forum explained what was happening. Basically the --no-site-packages flag removes the ones installed by webfaction for all users.

Next, I tried to setup the paths using a sitecustomize.py file. When python starts, it looks for that file. If it finds it, it imports it. Unfortunately that did not work either because webfaction already has one in /usr/local/lib/python2.5.

I tried adding an import statement to a .pth file. That solution did not work because there is no guarantee of when that file will be executed relative to the other .pth files. In my case, it ran before the paths I wanted to remove from sys.path.

For now, here's my soln. The only project I have running in virtualenv is django. I am just going to add an import statement in manage.py. It will configure the site path. On my development computer, I am not having all these problems. So I will put the import statement in a try block. If the config program is not found, it will just move on.

virtualenv and --no-site-packages

2010-08-27T14:41:00.000-07:00

I originally created my virtualenv without the flag --no-site-packages. I figured that the virtualenv paths would over-ride (come before) the global paths. That turned out to be wrong. The global paths come first.

One way to get around this is the create the env with the --no-site-packages flag and then manually add the global packages that I want using the virtualenvwrapper command add2virtualenv.

Since the env was already setup, I looked for ways to add that flag after the repo was created. Here is the answer. To have virtualenv not use global libs, just add the file "no-global-site-packages.txt" to the path: virtualenv_name/lib/python2.5 - Note this path is different from the one given on stackoverflow. I figured out the path by creating a test virtualenv and seeing where that file turned up.

But that's not really what I want to do. I want to have my env access the global paths because there are a lot of useful libs in there. It would be a lot of work manually adding each to the virtualenv.

Part of the answer seems to be in the pth files. It turns out that when you install a package in the virtual env using easy_install, easy_install creates a .pth file that removes versions of that package that are already on the path.

A quick peak in easy-install.pth reveals lines of code in addition to the paths! Here it is:

import sys; sys.__plen = len(sys.path)
./setuptools-0.6c11-py2.5.egg
./pip-0.7.2-py2.5.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

I found more info on this magic here. Apparently if you start a line with import and put all the code after it on the same line using ";" as a delimiter, then you can hack the path at startup.

Even More Setting Up Eclipse for Python and Django

2010-07-28T08:00:00.000-07:00

A while back, I punted on this effort. I cannot remember the details, but eventually things would get so buggered, that I would have to restart Eclipse. There was no discernible pattern. Everything would work fine, then my browser would stop seeing the Django dev server.

Re-starting Python processes did not help. Restarting the browser did not help. Switching browsers did not help. So I decided I do not need break points.

I am still using Eclipse. I like its auto complete and syntax checking. I like its user interface. But now I run the Django dev server in a command line window.

Error: pywintypes.py DLL load failed

2010-07-28T05:54:00.000-07:00

Some of my wxpython routines that used to work, stopped working. The error message was:

Traceback (most recent call last):

File "boot_com_servers.py", line 21, in

File "c:\python25\lib\site-packages\pythoncom.py", line 3, in

pywintypes.__import_pywin32_system_module__("pythoncom", globals())

File "C:\Python25\lib\site-packages\win32\lib\pywintypes.py", line 100, in __import_pywin32_system_module__

('.dll', 'rb', imp.C_EXTENSION))

ImportError: DLL load failed: The specified procedure could not be found.

If you look at pywintypes.py, you will see that it is a major hack that goes through all sorts of contortions to find the dlls. Here are some of the comments:

# This has been through a number of iterations. The problem: how to

# locate pywintypesXX.dll when it may be in a number of places, and how

# to avoid ever loading it twice. This problem is compounded by the

# fact that the "right" way to do this requires win32api, but this

# itself requires pywintypesXX.

# And the killer problem is that someone may have done 'import win32api'

# before this code is called. In that case Windows will have already

# loaded pywintypesXX as part of loading win32api - but by the time

# we get here, we may locate a different one. This appears to work, but

# then starts raising bizarre TypeErrors complaining that something

# is not a pywintypes type when it clearly is!

My hat is off to those that had the patience to deal with this BS.

The DLLs in question are:

pythoncom25.dll
pywintypes25.dll

Anyway - I added a few print statements to pywintypes.py and found that it was trying to load the dll from my recent installation of TortoiseHg. Those DLLs were in the TortoiseHg folder, but they were version 2.5.212.0. The version of those files in my python installation was 2.5.210.0. I checked the win32 download site and they are at version 2.5.214.0.

I was hoping this problem would be solved in the new version, so I installed it. That updated the DLLs in C:\WINDOWS\system32 and C:\Python25\Lib\site-packages\pywin32_system32 but the problem remained.

I tried deleting all the python related DLLs in TortoiseHg. This created a new error:

Traceback (most recent call last):

File "boot_com_servers.py", line 21, in

File "C:\Python25\lib\site-packages\pythoncom.py", line 2, in

import pywintypes

File "C:\Python25\lib\site-packages\win32\lib\pywintypes.py", line 124, in

__import_pywin32_system_module__("pywintypes", globals())

File "C:\Python25\lib\site-packages\win32\lib\pywintypes.py", line 61, in __import_pywin32_system_module__

raise ImportError("Module '%s' isn't in frozen sys.path %s" % (modname, sys.path))

ImportError: Module 'pywintypes' isn't in frozen sys.path

If TortoiseHg messes with the python DLL's then it probably causes lots of problems with other python programs. A quick google search shows that TortoiseHg also messes up PyScripter. It seems that problem can be eliminated by upgrading TortoiseHg to 0.8 or higher.

"The incompatibility between PyScripter and the TortoiseHg Mercurial addon has finally been fixed with the release of TortoiseHg 0.8, which replaces the Python Windows shell extensions with C++ based ones"

I am using TortoiseHg 0.7.6. The current version is 1.1.1. Since I almost never use TortoiseHg, I decided to uninstall it. Uninstalling it made the problem go away. Hopefully by the time I need to reinstall TortoiseHg, these problems will all be worked out.

***Update Nov 11,2010***

I have started using Hg again. This time it is for my own project. I like it much better than SVN. I reinstalled TortoiseHg 1.1.2. The problems described above did not reappear.

Hacking PDF with Python

2010-03-24T07:35:00.000-07:00

I recently did a project that involved extracting table data from some PDF's create by the UN. Here is an example. My first suggestion to my client was that he just contact the UN and see if he could get the data in some other format such as csv or excel. Much to my surprise, the UN had no interest in helping a college professor do some research - that no doubt will benefit humanity. So we were stuck with pdfs.

I had created pdfs with python before, but I had never extracted info from a pdf with python. So I did a quick google search and found the python package - pdfminer (when all went to hell later, subsequent searches showed that is the only game in town - for programs based in python). pdfminer outputs the results as text, html or xml. Great - certainly one of those formats would do the trick. So I told my client and we agreed that I should give it a shot.

My first approach was to convert to html. I envisioned tags such as table, tr, td, etc... Then I encountered the horror of pdf. As you may recall, pdf has been around for a long time. Long before the notion of semantic markup. As the name implies, the goal of "portable document format" is to make documents look EXACTLY the same on all platforms. Hence, pdf is more of a vector graphics format than information format. For example, if you have the letter "a" somewhere in a word, that is part of a sentence, pdf will specify the font, font-size and coordinates of the letter, but it might not know that the letter is part of a word which is part of a sentence. If you are a graphics designer, that is great, because it allows you to do all sorts of fine manipulations, such as kerning. But if you are trying to extract info from the document - it is a major headache. Also, pdf has no concept of a table. To pdf, a table is just a bunch of lines and letters.

So what does pdfminer give you when you request the output at html? A bunch of tags that attempt to define the position of the text - cannot remember the details because I switched to XML almost immediately. XML was not much better, but the structure was slightly easier to process.

At this point, I contacted the client, told him that things were much more complicated than I had first thought and asked him how he wanted to proceed. He suggested we try a commercially available program that some of his colleagues had some luck with. This seemed like a good idea to me. Thousands of people before me must have faced this problem. There must be hundreds of millions of pdfs on this planet by now. Why re-invent the wheel?

The program we chose was PDF Converter 6 by Nuance. It was about $50. Had a nice batch mode, so it was easy to convert all 125 pdf files into excel. Even better, it has a mode where it recognizes tables and will create an excel file where each sheet is a table. All the other clutter is filtered out! The biggest problem was that the excel sheets did not have the table titles.

At this point, the solution I attempted was to convert all the pdfs using PDF Converter. Then convert the pdfs again to text files using pdfminer. I would use regex to extract all the table titles and then apply them to the excel sheets.

This did not work for two reasons. First, the text in the file created by pdfminer was not in any particular order. To make matters worse, words and sentences were split up. It was a mess, with no info to put things back together again.

Second, PDF Converter is not very good at recognizing tables. Making things worse, it did not consistently fail on one particular table type. It was impossible to know which table it failed to find. Thus, for a pdf, I might get 8 sheets in the excel file but find 12 table titles. So it was not easy to figure out which title went with which sheet.

...urgh...months have passed...

I have not kept up the gory details of this post. Sorry. I wish I had, but it got too depressing. This project really ate my lunch. My client was very understanding. I probably could have bailed. But my (foolish) pride would not let me.

What I remember was that there were two sources of grief. First there was the pdf conversion software. Second, the files I was trying to process did not have consistent formatting. Also, the files often used spaces in numbers - one million - was 1 ooo ooo. This seriously confused the pdf converter.

In the end, I used a combination of python code and manual intervention. In retrospect, hiring a typist would have taken less time and been cheaper.

More setting up Eclipse for Django debugging

2010-03-23T12:41:00.000-07:00

I have been using eclipse with break points to debug a django app for a few days now. For a while the process was pretty flaky.

At first, I thought it was because I installed Aptana Web Development Tools after I had things working. The reason I installed that plugin was to get decent editors for HTML, CSS and javascript. After that install, I get the message: "Aptana JavaScript Scripting Console Started" whenever I start eclipse. It seems that message is benign.

I was able to get the debugger to work reliably with django by:

1) starting the remote debugger (as before)

2) using the Run icon to start manage.py instead of the Debug icon.

Setting up Eclipse for Python and Django

2010-03-16T13:06:00.000-07:00

After spending more hours than I would like to admit, I finally have Eclipse installed, with the Pydev plugin and I have debugging working for django. It was that last part that caused all the misery.

These notes are small additions to the excellent online tutorial by Nick Vlku. You will want to watch that first. I strongly recommend watching the whole thing once without pause. If you try to do things as he does them you will encounter errors - ones that he will give the solution to a little further into the screencast.

Here is my setup:

Windows XP

Django 1.1.1

Python 2.5

Eclipse 3.5.2

PyDev 1.5.5

This version of Pydev includes the extensions. Although the menus might be in different locations, it's not hard to guess what the new locations are. Enter the same values, etc... as Nick does.

After I got eclipse and pydev installed, I played around with it for a while with an ordinary python app (e.g non-django). I got used to the auto complete and debugging. Being familiar with these things definitely made it easier to setup debugging for django.

Here are some tips for setting up eclipse to work with the django development server. The goal is to have the development server work the same way it does outside of eclipse, but with breakpoints.

The first tip is to watch the windows process monitor for python.exe processes. It is very easy to end up with some orphaned processes which can cause things to stop working right. When in doubt, restart eclipse. After you start eclipse, you should see one python.exe process. Starting the remote debug server does not instantly start another python process. Running your code in debug mode adds two processes.

Next, Nick gives some code for your manage.py module. I could not get things to work using that code. I do not know if the problem was the code or something else. I eventually got things working using this code.

Next, make sure to start the Debug Server before you run your django code. When you run your django code in debug mode, you will see a message "pydev debugger: starting" - this is not the Debug Server. If you do everything right, you see three python.exe processes.

Here is what my debug window looks like when everything is working and the code is stopped at a breakpoint at the function homepage in the views.py module:

I have not figured out how to stop the debug server - outside of killing it from the process monitor. If you read the comments on Nick's post, you will see that others are wondering the same thing.

As for other setup tips, an other blogger recommended setting the environment variables DJANGO_SETTINGS_MODULE and PYTHONPATH in the Python Run configuration. I found that made things worse. If you do not know what I am talking about, then don't worry about it.

Others have suggested adding the --noreload flag after the "runserver" command in the Run Python configuration. Evidently setting that flag causes eclipse to print more informative messages in the console (like the django development server run outside of eclipse) with the downside being that you must stop and start the server if you change any of your code. For me the most important thing to know was that django debugging can work with or without that flag.