Profiling and Optimizing Python Code

All programmers have heard the advice: "Don't prematurely optimize code." What exactly does that mean? It means that you shouldn't guess about what is causing your code to run slowly. Chances are you'll guess wrong. Instead of guessing, use profiling and benchmarking tools to quickly and accurately identify the performance bottlenecks in your scripts.

Every now and then I run into a piece of Python code that just doesn't run as fast as I would like. I use 3 different Python tools to find and fix Python performance problems.

cProfile:

cProfile is a module that is included in the Python Standard Library. It logs function calls and execution times. There is also a pure-python version named 'profile'.

KCachegrind:

KCachegrind was written to visualize the output generated by Callgrind (a C profiler), but the function call logs from cProfile can be converted to the KCachegrind format with this script. KCachegrind uses the KDE framework. On Linux boxes, KCachegrind is usually bundled in a package with other development tools written for KDE. The package is named 'kdesdk' or something similar.

timeit:

timeit is a module included in the Python Standard Library that is used for measuring the execution time of arbitrary pieces of code.

Here is the code that needs to be improved:

class LameStringBuilder(object):
    def __init__(self, cols=40, rows=10000):
        self.cols = cols
        self.rows = rows

    def build(self, val):
        built = ''
        for i in range(self.rows):
            built += self.build_row(val) + "\n"
        return built

    def build_row(self, val):
        built = ''
        for i in range(self.cols - 1):
            built += val + ','
        built += val
        return built

Here is the code to execute a profile test and format the results into something that KCachegrind can understand:

import cProfile

# Use this module to convert profile data
# into the KCacheGrind format.
#
# The module is available here:
# http://www.gnome.org/~johan/lsprofcalltree.py
import lsprofcalltree

import lame_string_builder

def test():
    """Code to profile."""
    l = lame_string_builder.LameStringBuilder(40, 10000)
    l.build('foo')

if __name__ == "__main__":
    # Profile function
    p = cProfile.Profile()
    p.run('test()');

    # Get profile log in KCacheGrind format
    # and dump to file.
    k = lsprofcalltree.KCacheGrind(p)
    out = open('lame_profile.kgrind', 'w')
    k.output(out)
    out.close()

After running the test, launch KCachegrid and open the profile file to view the results.



The 'Flat Profile' and 'Callee Map' are the 2 most useful displays for finding the bottle necks in your code.

The 'Flat Profile' describes each function called in the test with the following columns:

• Incl. - The total percentage of time spent within a function.


• Self - The percentage of time spent within a function NOT including inner function calls.


• Called - The total number of times the function was called during the test.


• Function - The name of the function being described.

The 'Callee Map' represents the different functions called during the test. Each function is drawn as a rectangle. Inner function calls are drawn on top of their parent function's rectangle. The area used to draw each function is proportional to the execution time for each function call relative to the total execution time of the parent (also printed as a percentage in the graph).

In the example above it is easy to pick out where the bottle necks are by looking at the 'Callee Map'. Most of the test time was spent within the 'LameStringBuilder.build_row' method, so it is the main bottle neck. Other significant sources of time include calls to 'LameStringBuilder.build' and the built-in function 'range'.

After identifying the bottlenecks, I created a better version of the 'LameStringBuilder' class:

import lame_string_builder

class LessLameStringBuilder(lame_string_builder.LameStringBuilder):
    def build(self, val):
        return "\n".join([self.build_row(val) for i in xrange(self.rows)])

    def build_row(self, val):
        return ",".join([val for i in xrange(self.cols)])

After making changes to the code, Python's built-in 'timeit' module can be used to simply and accurately measure the differences in execution speed between the old and the new versions:

import timeit

import lame_string_builder
import less_lame_string_builder

def test_lame(val, cols, rows):
    l = lame_string_builder.LameStringBuilder(cols, rows)
    l.build(val)

def test_less_lame(val, cols, rows):
    l = less_lame_string_builder.LessLameStringBuilder(cols, rows)
    l.build(val)

def test_generic(repeat, name, args):
    print "%s (x%i):" % (name, repeat)
    t = timeit.Timer("%s(%s)" % (name, args), "from __main__ import %s" % name)
    print t.timeit(repeat)

def test_all(repeat, val, cols, rows):
    args = "'%s', %i, %i" % (val, cols, rows)   
    
    for name in ("test_lame", "test_less_lame"):
         test_generic(repeat, name, args)

if __name__ == "__main__":
    test_all(100, 'foo', 40, 10000)

The new code is significantly faster!

Role based security with Python

Most moderately complex enterprise applications require some sort of role based security and access control. For example, an employee should have access to fill-out her time card, but a manager should also be able to approve the time card. This post outlines how to create an 'access control list' (acl) for Python.

The acl is an object that can be queried to determine if a particular role has permission to access a resource. Each permission is also associated with a 'privilege'. For example, a manager may have both the 'read' and 'write' privileges for the weekly schedule, but an employee only has the 'read' privilege.

import acl

# Each user name should be associated with a role.
role = get_role(username)

# Query the acl to see if a role has access to a resource.
# params: role name, resource, privilege
acl = acl.Acl()
if not acl.check_access(role, 'time_card', 'write'):
    # User doesn't have access to this resource!!
    pass

The Acl class supports role inheritance. If the 'manager' role inherits the 'employee' role, then managers will have all of the same permissions as employees.

# Build acl list
acl = acl.Acl()

employee = acl.Role('employee')
resource = acl.Resource('time_card')
resource.set_privilege('write')
employee.set_resource(resource)
acl.set_role(employee)

manager = acl.Role('manager')
manager.set_parent(employee)
resource = acl.Resource('time_card')
resource.set_privilege('approve')
manager.set_resource(resource)
acl.set_role(manager)

if acl.check_access('manager', 'time_card', 'write'):
    # YES!
    pass

if acl.check_acces('manager', 'time_card', 'approve'):
    # YES!
    pass

if acl.check_access('employee', 'time_card', 'write'):
    # YES !
    pass

if acl.check_access('employess', 'time_card', 'approve');
    # NO!
    pass

I'm normally not a very big fan of XML, but in this particular case it is a good format to use if you prefer to store your acl in a configuration file. The Acl object's 'build_acl' method populates the object from a XML file with the following format:

<?xml version="1.0"?>
<!--
- This file configures the roles (user groups)
- and permissions for accessing the system.
-->
<config>
    <!--
    - Setup roles here.
    - Use the 'inheritFrom' tag to
    - inherit permissions from another role.
    -->
    <roleSet>   
        <role>
            <name>customer</name>
        </role>
        <role>
            <name>employee</name>
            <inheritFrom>customer</inheritFrom>
        </role>
        <role>
            <name>manager</name>
            <inheritFrom>employee</inheritFrom>
        </role>
    </roleSet>

    <!--
    - Set permissions for accessing application components here.
    - resource -> property being access controlled.
    - role -> group or user that can access resource.
    - privilege -> privilege that role can use with resource.
    -
    - Each permission tag can contain multiple
    - resources, roles, and privileges.
    -->
    <permissions>
        <permission>
            <resources>
                <resource>contact_details</resource>
                <resource>profile</resource>
            </resources>
            <roles>
                <role>customer</role>
            </roles>
            <privileges>
                <privilege>read</privilege>
                <privilege>write</privilege>
            </privileges>
        </permission>

        <permission>
            <resources>
                <resource>time_card</resource>
            </resources>

            <roles>
                <role>employee</role>
            </roles>
            <privileges>
                <privilege>read</privilege>
                <privilege>write</privilege>
            </privileges>
        </permission>
        
        <permission>
            <resources>
                <resource>time_card</resource>
            </resources>
            <roles>
                <role>manager</role>
            </roles>
            <privileges>
                <privilege>approve</privilege>
            </privileges>
        </permission>

    </permissions>
</config>

Here is the code for the acl.py module:

"""Role based security"""
from xml.dom.minidom import parse

class AccessError(Exception):
    pass

class Resource(object):
    """An Resource is an object that can be accessed by a Role."""
    def __init__(self, name=''):
        self.name = name
        self._privileges = {}

    def set_privilege(self, privilege, allowed=True):
        self._privileges[privilege] = allowed

    def has_access(self, privilege):
        if privilege in self._privileges:
            return self._privileges[privilege]
        return False
    
    def __str__(self):
        rpr = self.name + ': '
        for privilege, access in self._privileges.iteritems():
            rpr += "%s:%s " % (privilege, access)
        return rpr

class Role(object):
    def __init__(self, name=''):
        """An Acl role has access to resources with specific privileges."""
        self.name = name
        self._parents = {}
        self._resources = {}

    def set_parent(self, parent):
        self._parents[parent.name] = parent

    def set_resource(self, resource):
        self._resources[resource.name] = resource

    def has_access(self, attr_name, privilege):
        if attr_name in self._resources:
            if self._resources[attr_name].has_access(privilege):
                return True

        for parent in self._parents.values():
            if parent.has_access(attr_name, privilege):
                return True

        return False
    
    def __str__(self):
        rpr = self.name + ":\n"
        rpr += "parents:\n"
        for parent in self._parents.keys():
            rpr += "\t%s\n" % parent
        rpr += "resources:\n"
        for resource in self._resources.values():
            rpr += "\t%s\n" % resource.describe()
        return rpr

class Acl(object):
    """Manages roles and resources.
    
    Singleton class.
    """

    class __impl:
        """Implementation of the singleton interface"""
        
        def __init__(self):
            self._acl = {}

        def set_role(self, role):
            self._acl[role.name] = role
            
        def check_access(self, role_name, resource, privilege):
            """Check whether a role has access to a resource or not."""

            if not role_name in self._acl:
                raise AccessError('Role does not exist.')
            return self._acl[role_name].has_access(resource, privilege)

        def build_acl(self, file): 
            """Build acl from an XML file."""

            self._acl = {}
            roles_to_create = {}
            dom = parse(file)
            
            # Find roles to create
            roles_nodes = dom.getElementsByTagName('roleSet')
            for roles_node in roles_nodes:
                role_nodes = roles_node.getElementsByTagName('role')
                for role_node in role_nodes:
                    name_nodes = role_node.getElementsByTagName('name')
                    parent_nodes = role_node.getElementsByTagName('inheritFrom')
                    role_name = name_nodes[0].childNodes[0].data
                    roles_to_create[role_name] = []

                    # Find role parents
                    for parent_node in parent_nodes:
                        roles_to_create[role_name].append(parent_node.childNodes[0].data)

            # build inheritence chain
            for role, parents in roles_to_create.iteritems():
                self.set_role(self._create_role(role, roles_to_create))

            # assign permissions
            permissions = dom.getElementsByTagName('permissions')
            for permissions_node in permissions:
                permission_nodes = permissions_node.getElementsByTagName('permission')
                for permission_node in permission_nodes:
                    resource_nodes = permission_node.getElementsByTagName('resource')
                    role_nodes = permission_node.getElementsByTagName('role')
                    privilege_nodes = permission_node.getElementsByTagName('privilege')

                    for resource_node in resource_nodes:
                       resource = Resource()
                       resource.name = resource_node.childNodes[0].data
                       for privilege_node in privilege_nodes:
                           resource.set_privilege(privilege_node.childNodes[0].data)

                       for role_node in role_nodes:
                           try:
                               role = self._acl[role_node.childNodes[0].data]
                           except:
                               raise AccessError('Role in permission is not defined.')

                           role.set_resource(resource)

        def _create_role(self, role_name, roles_to_create):
            """Recursively create parent roles and then create child role."""
            
            if role_name in self._acl:
                role = self._acl[role_name]
            else:
                role = Role()
                role.name = role_name
                
            for parent_name in roles_to_create[role_name]:
                if parent_name in self._acl:
                    parent = self._acl[parent_name]
                else:
                    parent = self._create_role(parent_name, roles_to_create)
                    self.set_role(parent)
                role.set_parent(parent)
            return role
        
        def __str__(self):
            rpr = ''
            for role in self._acl.values():
                rpr += '----------\n'
                rpr += role.describe()
            return rpr

    __instance = None

    def __init__(self):
        """ Create singleton instance """
        
        # Check whether an instance already exists.
        # If not, create it.
        if Acl.__instance is None:
            Acl.__instance = Acl.__impl()

        self.__dict__['_Acl__instance'] = Acl.__instance

    def __getattr__(self, attr):
        """ Delegate get access to implementation """

        return getattr(self.__instance, attr)

    def __setattr__(self, attr, val):
        """ Delegate set access to implementation """

        return setattr(self.__instance, attr, val)

SQL Workshop

I've recently finished a SQL tutorial for a relational database workshop I'll be teaching. The tutorial covers basic SQL queries with SQLite and MySql, as well as basic database creation with SQLite. The tutorial is very basic, and the examples are geared toward biologists who need to use a database for performing research.

Facebook's Tornado

Today I got my first chance to play with Facebook's Python based web server and framework Tornado. The framework is single threaded and asynchronous, similar to how the Twisted framework operates, and Facebook uses the technology to provide data to the 'Friend Feed'.

Python thread creation and maintenance have relatively high overhead, so asynchronous solutions can provide better performance and scalability than more traditional threaded server implementations. This is especially true for servers with high numbers of concurrent connections, and for long-polling and streaming connections that spend most of their time waiting around for a message to be published to a client. Tornado's benchmarks look impressive against threaded Python web servers, but I wonder why they didn't include Twisted, their closest competitor, in the benchmarking.

My first impression is that Tornado is much easier to use than Twisted, but also much less powerful. Twisted is a monolithic package that can be used with many different networking protocols, but Tornado is focused only on http. Tornado provides built in support for templating, authentication, and signed cookies. One really cool feature Tornado provides is semi-automatic XSRF protection, which should save time for developers who would otherwise need to manually implement countermeasures.

The documentation for Tornado is very slim. For example, I couldn't find any information about how to interact directly with Tornado's main event loop (IOLoop). Fortunately, Tornado's code base is small and readable, so reading the source will quickly get you up to speed.

While implementing a Tornado channel for the AmFast project, I ran into a problem that I have also encountered with Twisted. Both Tornado and Twisted use callbacks to complete a request. As an example, if a long-poll client is waiting for a message, you call RequestHandler.finish() to send a response back to the client when a message is published.

The above method works well for a single server instance, but what about when you have multiple servers behind a proxy? A server process may publish a message for a client that is connected to an entirely different server process. The publishing process has no way to notify the subscribing process that a client has received a message.

This problem can be solved by polling a database table or a file that is accessible to all server processes, but that takes system resources, especially if you have many clients and your polling interval is short. One of my future goals is to figure out a more elegant way for the publishing server process to notify the subscribing server process when a client receives a message.

AmFast adds support for Google App Engine and Django

I just released version 0.4.0 beta of AmFast, a Flash remoting package for Python. The new release includes support for Google App Engine and the Django framework.

Browse the code here.


Check out the live Google App Engine demo.

To get the code:

svn checkout https://amfast.googlecode.com/svn/tags/0.4.0b

YAPT (Yet Another Python Tutorial)

The tutorial I created for the Python workshop is posted here. The tutorial is geared towards bioinformatics users, and includes molecular biology related exercises.

I created the tutorial with the Sphinx documentation package, which turned out to be the perfect tool for the job.

Learn Python Workshop

I'm going to be teaching an intro to Python workshop next month. It will consist of 3 3 hour days and will cover the basics of Python. Dates are June 15th, 17th, 19th. The workshop is free for all U of A students, staff, and faculty. I have never lead a workshop before, so it should be a lot of fun.

Register for the Python workshop

Real-time Flex messaging with AmFast

This example is very similar to the previous, but we're going to use AmFast's HTTP streaming channel to implement real-time flex messaging. Get the complete example code here.

First let's setup the Producer and Consumer in Actionscript.

// Configure the ChannelSet that
// messages will be sent and received over.
import mx.messaging.ChannelSet;

// To use HTTP streaming, the
// StreamingAMFChannel class must be used.
import mx.messaging.channels.StreamingAMFChannel;
var channelSet:ChannelSet = new ChannelSet();
var channel:StreamingAMFChannel = new StreamingAMFChannel("channel-name", "server-url");
channelSet.addChannel(channel);

// Setup a Consumer to receive messages.
import mx.messaging.Consumer;
var consumer:Consumer = new Consumer();

// The consumer's destination is the 'topic'
// name that the consumer will be subscribed to.
// The consumer will receive all messages
// published to the topic.
consumer.destination = "topic";

// Use the ChannelSet that was already created.
consumer.channelSet = channelSet;

// This event listener will be called whenever
// the consumer receives a message from the server.
consumer.addEventListener(MessageEvent.MESSAGE, newMsgHandler);

// The consumer won't start receiving messages
// until it is subscribed.
consumer.subscribe();

// Setup a Producer to publish messages.
import mx.messaging.Producer;
var producer:Producer = new Producer();
producer.destination = "topic";
producer.channelSet = channelSet;

// Create an Async message and send it
// to all other clients subscribed to the topic.
import mx.messaging.messages.AsyncMessage;
var msg:AsyncMessage = new AsynMessage();

// Set the message's body attribute to the
// object that is going to be published.
//
// In this case the object contains
// the X and Y coordinates of a Sprite.
msg.body = {'x': 10, 'y': 10};
producer.send(msg);

Next we'll setup the server.

# Create a ChannelSet to serve messages.
from amfast.remoting.channel import ChannelSet
from amfast.remoting.wsgi_channel import StreamingWsgiChannel
channel_set = ChannelSet()

# Each individual ChannelSet can use
# one or more Channels. When messages
# are published through the ChannelSet,
# they will be published to all subscribed clients,
# regardless of which Channel the clients
# are connected through.

# Create a HTTP streaming channel.
#
# When a client connects to a streaming channel,
# the HTTP connection remains open until the client
# disconnects. When messages are published,
# they are immediately dispatched to clients connected
# to HTTP streaming channels.
stream_channel = StreamingWsgiChannel('stream-channel')

# WsgiChannels objects are wsgi applications
# that can be served with any Wsgi server.
# CherryPy is being used in this example.
cherrypy.tree.graft(stream_channel, '/amf')

# Start the server.
# App() is your root controller class
# for non-AMF functions.
cherrypy.quickstart(App(), '/')

# That's it, our server is up and running.
# It's that simple.

To test the example, download the full code from the repo. To run the example you'll need to install AmFast and CherryPy or Twisted 1st.

# To server the example with cherrypy
python cp_server.py

# To server the example with Twisted
twistd -noy twisted_server.tac

Open two browser windows and browse to 'http://localhost:8000'. Click the 'Subscribe' button in both windows. In the 1st window click on the 'Master' button. In the 1st window, click on the yellow circle, and drag the circle around the screen. As the circle is dragged around the screen, the position of the circle will be updated in the 2nd window. Pretty cool stuff.

Create a chat client and server with Flex and Python

In this example we'll create a simple Flex chat client that uses Producer/Consumer messaging to send and receive messages through a Python server. The server will use AmFast's HTTP polling and long-polling channels. Get the complete example code here.

First let's setup the Producer and Consumer in Actionscript.

// Configure the ChannelSet that
// messages will be sent and received over.
import mx.messaging.ChannelSet;
import mx.messaging.channels.AMFChannel;
var channelSet:ChannelSet = new ChannelSet();
var channel:AMFChannel = new AMFChannel("channel-name", "server-url");
channelSet.addChannel(channel);

// Setup a Consumer to receive messages.
import mx.messaging.Consumer;
var consumer:Consumer = new Consumer();

// The consumer's destination is the 'topic'
// name that the consumer will be subscribed to.
// The consumer will receive all messages
// published to the topic.
consumer.destination = "topic";

// Use the ChannelSet that was already created.
consumer.channelSet = channelSet;

// This event listener will be called whenever
// the consumer receives a message from the server.
consumer.addEventListener(MessageEvent.MESSAGE, newMsgHandler);

// The consumer won't start receiving messages
// until it is subscribed.
consumer.subscribe();

// Setup a Producer to publish messages.
import mx.messaging.Producer;
var producer:Producer = new Producer();
producer.destination = "topic";
producer.channelSet = channelSet;

// Create an Async message and send it
// to all other clients subscribed to the topic.
import mx.messaging.messages.AsyncMessage;
var msg:AsyncMessage = new AsynMessage();

// Set the message's body attribute to the
// object that is going to be published.
//
// In this case the object is a String,
// but the message body can be any
// type of object.
msg.body = "This is being published!";
producer.send(msg);

Next we'll setup the server.

# Create a ChannelSet to serve messages.
from amfast.remoting.channel import ChannelSet
from amfast.remoting.wsgi_channel import WsgiChannel
channel_set = ChannelSet()

# Each individual ChannelSet can use
# one or more Channels. When messages
# are published through the ChannelSet,
# they will be published to all subscribed clients,
# regardless of which Channel the clients
# are connected through.

# Create s standard polling channel.
polling_channel = WsgiChannel('poll-channel')

# A long-polling channel is created similar
# to a normal channel, but the wait_interval
# argument must be set to -1.
long_channel = WsgiChannel('long-channel', wait_interval=-1)

# WsgiChannels objects are wsgi applications
# that can be served with any Wsgi server.
# CherryPy is being used in this example.
cherrypy.tree.graft(polling_channel, '/amf')
cherrypy.tree.graft(long_channel, '/longPoll')

# Start the server.
# App() is your root controller class
# for non-AMF functions.
cherrypy.quickstart(App(), '/')

# That's it, our server is up and running.
# It's that simple.

To test the example, download the full code from the repo. To run the example you'll need to install AmFast and CherryPy or Twisted 1st.

# To server the example with cherrypy
python cp_server.py

# To server the example with Twisted
twistd -noy twisted_server.tac

Open two browser windows and browse to 'http://localhost:8000'. In the 1st window set the messaging url to 'http://localhost:8000/amf' to use the polling channel. In the 2nd window set the messaging url to 'http://localhost:8000/longPoll' to use the long-polling channel. Click the 'Subscribe' button in both windows. Enter a message in the message entry box, and press the 'Publish' button to publish the message. The published message will appear in both clients.

'Pushing' messages to Flex clients with AmFast

One of the coolest aspects of Flex development is the ability to 'push' messages from the server to clients. With the newest release of AmFast, you can finally utilize this capability with all of the goodness of Python. This will be the 1st of 3 posts describing how you can use the AmFast Flash remoting package to push messages to Flex clients from a Python server.

AmFast supports pushing messages to clients over HTTP using 3 different strategies: polling, long-polling, and streaming. Here is a good article describing the differences between those strategies. Go ahead and skip past the XML configuration sections, since you won't need any XML to configure AmFast :)

Streaming channels deliver messages in real time, but also consume the most server resources. Standard polling consumes the least amount of server resources, but is also the slowest of the three strategies. Long-polling sits in the middle and offers a nice compromise between latency and resource consumption.

AmFast has built-in support for using any of the 3 messaging strategies with either a multi-threaded WSGI server, or the Twisted web framework. One thing to think about when implementing a multi-threaded server is that any client using long-polling or streaming may consume a thread for a significant amount of time, even if all it's doing is waiting for a new message. I haven't done any tests, but I'm guessing that Twisted's single-threaded, asynchronous design can handle more concurrent connections then a multi-threaded implementation.

On the client side, the Actionscript Consumer class is used to receive messages from a server, and the Producer class is used to send messages to a server. Any type of object can be pushed to clients, so the same technology can be used to build many different types of applications.

In the next post I'll explain the details of using AmFast's HTTP polling and HTTP long-polling channels to create a simple chat client. In the following post I'll show an example of using HTTP streaming to create an app where a master client can manipulate a graphical entity on the screen, and changes are replicated to all other subscribed clients.

AmFast 0.3.0 Released!

The latest version of AmFast is out. In addition to the changes listed below, the documentation has been updated, and there are several new examples.

remoting

• Producer/Consumer messaging via HTTP polling.
• Channels and ChannelSets allow resources to be exposed in multiple ways.
• Built in Channels for CherryPy, Twisted Web, and straight-up WSGI.
• Easy to configure authentication for NetConnection and RemoteObject.
• Connection objects hold session attributes tied to Flex client id.
• Customizable Endpoints to encode/decode messages with external encoding/decoding libraries.
• PyAmfEndpoint to encode/decode messages with pure Python, with no C compiler dependencies.

encode/decode

• Read and write to file-like-objects or strings (using strings is much faster).
• Automatic type conversion.
• amfast.class_def.as_types.AsProxy and amfast.class_def.as_types.AsNoProxy allow overriding 'use_collections' and 'use_proxies' settings for a individual objects.
• readExternal and writeExternal methods of the IEXTERNALIZABLE interface accept different parameters to make custom encodings easier.
• amfast.encoder.Encoder and amfast.decoder.Decoder: Wrappers around amfast.encode and amfast.decode, so you don't need to pass the same arguments every time you encode or decode.

Plater 0.1.3 maintenance release

Plater 0.1.3 (laboratory information management package) maintenance release today. The release contains a bug fix that was affecting parsing of CSV plate maps.

AmFast 0.3 release is coming soon

I've been working on AmFast 0.3 for a while now, and it looks like I should be able to release it next weekend. 0.3 has cool new features such as: Producer/Consumer messaging, pure-Python encoding and decoding using PyAmf, and a bunch of other cool stuff.

The coding is done, and I'm working on documentation, testing, and a couple of new example applications.

AmFast 0.2.2 Released!

The latest version of AmFast is out. Improvements include an Actionscript code generator, an example using the Twisted framework, and several bug fixes.

Flash and Python Presentation

On Friday I presented information about Flash and Python to several developers from iPlant.

The presentation covered the current state of Python tools designed to communicate with various flavors of Flash. I will be giving the same presentation to TuPLE this coming Tuesday.

Here are the slides.

AmFast 0.2 Released!

Check out the latest version of AmFast. The 0.2 version implements AMF0, AMF3 and supports configurable remoting with NetConnection and RemoteObject.

It's fast, flexible, and easy to use.

AmFast Released!

I released the alpha version of AmFast today. AmFast is an AMF3 encoder/decoder Python extension.

Some un-sophisticated testing shows that AmFast is around ~18x quicker than PyAmf. 10,000 runs through the test_complex_encode_decode_dict() unit test takes ~105 seconds on my machine using PyAmf, and ~5.8 seconds using AmFast.

If you want to try it out, you can download the package from PyPi, and take a look at the meager documentation.

AmFast supports all of the data types that PyAmf does, but there is not yet any functionality for implementing remoting, or working with AMF0. I'm hoping that AmFast will be able to integrate into PyAmf, so that you can optionally use either the existing pure Python AMF3 encoder/decoder, or AmFast within the PyAmf framework.

This was my first time working with the Python C API, and I think it's pretty cool. Besides having to remember to increment and decrement Python object reference counts in the right spots, it was pretty easy to learn. I'm not anywhere near a C expert, and I wrote this extension in my spare time over the course of a couple of weeks.

Plater Released!

I released Plater 0.1 today.

Plater is a Python package that contains common LIMS classes that can be used to build a custom LIMS application.

Instead of starting your LIMS application from scratch, use the Plater package to represent common LIMS domain models, and piece them together in whatever configuration your application requires.

Take a look at the Plater project page for details on how to download/install/use Plater.

AMF vs. JSON

You can find many blog posts comparing AJAX data transfer formats JSON and XML to AMF3, but they all seem to miss what I see as the 3 biggest advantages to AMF3.

1: AMF3 supports references for objects, strings, and class definitions. If you're encoding an object graph that contains references to the same object over and over, or even the same object type over and over, AMF3 will be significantly smaller over the wire, and once you get the data you won't need to do any manually reconciling of objects that are really the same thing.

2: AMF3 can decode typed objects that have methods, derived properties, and a static class definition. That saves time because you have to do less post-parsing processing of the data. It also makes development a lot quicker, especially if you're using an ORM on the server side. You can send objects retrieved from the database straight over to the client, the user can modify the objects with the GUI, and the objects can be sent straight back to the server to save changes. Your server-side AMF encoder/decoder library and ORM library take can care of all the CRUD grunt work, so all you have to do is figure out a decent data model and how to represent it in the client GUI. I use Python and PyAMF on the server side, which even has support for lazy-loading attributes of objects persisted with the SQLAlchemy ORM.

3: You can implement the IExternalizable interface on any AS3 class to fully customize the object encoding/decoding process. If you're brave and need optimization for large data sets, you can even write and read directly to/from the AMF3 byte stream and use your own custom encoding scheme suited for your data type

PyAMF SQLAlchemy Adapter

PyAMF 0.4 was recently released, and the new code contains SQLAlchemy support by default, plus a bunch of other cool new features and bug fixes.

Check out my example Flex/PyAMF/SQLAlchemy project for ideas of how to get started with PyAMF and SQLAlchemy.