Saturday, March 09, 2013

How do you set up for multi package Python development?

Or, how do I use my models outside of a Pyramid application?

I've developed a little Pyramid app as a mobile backend version of FuelMyRoute.com. As natural, I've defined the database models inside the application. But, now I want to develop a new gas price data importer piece of code that will live outside of the Pyramid web application. This importer will need to make use of the same models as defined in the Pyramid app. So what to do?

Well, ideally I would have the models defined outside of the Pyramid app in its own package. So I would end up with three packages: the Pyramid app is a package, the models are a package, making the importer code a third package. The importer package would depend on the models package, just like the Pyramid app.

Now, in Python there are lots of mechanisms to pull this off. First we could just manipulate the sys.path in each package to pull in the needed dependencies. Second we could set a PYTHONPATH to pull in the other packages. But the way I decided to do it was to install the packages into a virtual environment. I got this idea from the Pyramid tutorial/documentation where it has you do a python setup.py develop into the virtual environment for the app. python setup.py develop is basically a way to symlink in your package into the system path of the virtual environment.

Here are the three packages. Each package name is prefixed with gptp-, which is just an codename for an earlier version of FuelMyRoute.com.

  1. gptp-models - Defines the SQLAlchemy models
  2. gptp-pyramid - The Pyramid application, depends on gptp-models
  3. gptp-importer - Gas price data importer code, depends on gptp-models

Here's the setup.py file for the gptp-models package:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from setuptools import setup, find_packages

# http://pythonhosted.org/distribute/setuptools.html#basic-use
setup(
    name='gptp-models',
    version='0.1dev',
    packages=find_packages(),
    install_requires=[
        # <0.8 includes stupid stuff like 0.8b2, so have to defensively prevent
        # alphas, betas etc. http://stackoverflow.com/a/14405269/1419499
        'SQLAlchemy<0.7.99',
        #... additional dependecies
        ]
)

This is just about as barebones as it can get for a setup.py. I believe that only name, version and packages are required. This package depends on SQLAlchmey, amongst other things, so I've included that to illustrate how to declare dependencies.

Here's the setup.py for gptp-pyramid:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
import os

from setuptools import setup, find_packages

here = os.path.abspath(os.path.dirname(__file__))
README = open(os.path.join(here, 'README.txt')).read()
CHANGES = open(os.path.join(here, 'CHANGES.txt')).read()

requires = [
    'pyramid<=1.3.99',
    'SQLAlchemy<=0.7.99',
    'transaction',
    'pyramid_tm',
    'pyramid_debugtoolbar',
    'zope.sqlalchemy',
    'waitress',
    'gptp-models',
    # .. more dependencies ..
    ]

setup(name='gptp',
      version='0.0',
      description='gptp',
      long_description=README + '\n\n' +  CHANGES,
      classifiers=[
        "Programming Language :: Python",
        "Framework :: Pylons",
        "Topic :: Internet :: WWW/HTTP",
        "Topic :: Internet :: WWW/HTTP :: WSGI :: Application",
        ],
      author='',
      author_email='',
      url='',
      keywords='web wsgi bfg pylons pyramid',
      packages=find_packages(),
      include_package_data=True,
      zip_safe=False,
      test_suite='gptp',
      install_requires=requires,
      entry_points="""\
      [paste.app_factory]
      main = gptp:main
      [console_scripts]
      initialize_gptp_db = gptp.scripts.initializedb:main
      """
      )

There's more going on here, but most of the extra stuff is stuff that was auto-generated by Pyramid. The important thing is that on about line 18 the package gptp-models is listed as a required dependency.

Finally we have gptp-importer's setup.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
from setuptools import setup, find_packages

# http://pythonhosted.org/distribute/setuptools.html#basic-use
setup(
    name='gptp-importer',
    version='0.1dev',
    packages=find_packages(),
    install_requires=[
        'gptp-models'
        ]
)

About what you might expect at this point.

So now with my three packages set up, I can proceed with setting up a virtual environment. I'm using Python 3.3 so I set up the virtual environment with the built in pyvenv command and install distribute and pip into it, using this script, named bootstrap-pyvenv:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
#!/bin/sh

# Create Python virtual environment
pyvenv $1

# install distribute and pip
curl -O http://python-distribute.org/distribute_setup.py
$1/bin/python distribute_setup.py
$1/bin/easy_install pip

# Clean up
rm distribute*

So I run:

bootstrap-pyvenv gptpenv

Then, activate the environment and run python setup.py develop in each package, starting with gptp-models of course:

source gptpenv/bin/activate
cd gptp-models
python setup.py develop
cd ../gptp-importer
python setup.py develop
cd ../gptp-pyramid
python setup.py develop

Okay, so that's set up. But how do I get Pyramid to use my models that are defined in a separate package? Here's what my models.py looked like before I made my changes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
from sqlalchemy import (
    Column,
    DateTime,
    Float,
    ForeignKey,
    Integer,
    String,
    types,
    )

from sqlalchemy.ext.declarative import declarative_base

from sqlalchemy.orm import (
    relationship,
    scoped_session,
    sessionmaker,
    )

from zope.sqlalchemy import ZopeTransactionExtension

from datetime import datetime
from shapely.wkb import loads
from struct import pack, unpack

from geo.proj import ProjectedPoint, utm_projector

DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
Base = declarative_base()

# ... start defining my models here ...

I moved my models into gptp-models/models.py and updated the pyramid models.py to be:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
import models
from sqlalchemy.orm import (
    scoped_session,
    sessionmaker,
    )

from zope.sqlalchemy import ZopeTransactionExtension

DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
Base = models.Base

The import models imports my models from the gptp-models package (could probably be namespaced better, but this suffices for now). The line

1
Base = models.Base

is just a convenience so that I can continue to import Base from the Pyramid models.py.

That's basically it. I just need to update the imports elsewhere in the app. For example, in my views.py I had

1
2
3
4
5
6
7
# ...
from .models import (
    DBSession,
    GasStation,
    GasPrice,
)
# ...

which becomes:

1
2
3
4
5
6
7
8
9
# ...
from .models import (
    DBSession,
)
from models import (
    GasStation,
    GasPrice,
)
# ...

Other Approaches to the Multiple Python Packages Problem

I asked about what people do in this situation on Twitter:

Pradeep Gowda replied with

I think that would probably be the better way to go for a larger project and multiple developers. For what I'm doing, since it's just me and a side project, the setup described above is sufficient.

Pradeep shared the following benefit of going the local pypi approach:

He also shared this link on how to set up a local PyPI.

So that's another option that is worth considering too.

Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)