Derrick Petzold

Django logging, json, and syslog

Views: 97 Comments: 0

Tags:

Posted January 12, 2012 by derrick

In this post I will show you how to configure logging so your application servers can send info in nicely parseable json format to a centralized server for analysis.

So first things first. Lets begin by moving the logging out of settings.py and into it own config. Why because its easier to read and edit in yaml than in python/json and it shows up great in vim/emacs. So let's replace this:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse',
        }
    },
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        }
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
    }
}

with this

import yaml
SITE_ROOT = os.path.dirname(os.path.realpath(__file__))
LOGGING = yaml.load(open(SITE_ROOT + '/logging.yaml', 'r'))

and put this into logging.yaml

version: 1
disable_existing_loggers: True
        
filters:
    require_debug_false:
        (): django.utils.log.RequireDebugFalse

handlers: 
    mail_admins:
        level: ERROR
        filters: [require_debug_false]
        class: django.utils.log.AdminEmailHandler

loggers:
    django.request:
        handlers: [mail_admins]
        propagate: True
        level: ERROR

Yay. Now we have a logging config that's easy to read and edit. To make the logging as painless as possible use Madzak's excellent module python-json-logger. I went the distance and defined my own _fmt parser.

class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def parse(self):
        return eval(self._fmt)

That allows for the format to be defined as python string. Append the json formatter, syslog handler and the app logger to their respective locations in logging.yaml.

formatters:
    (): log.CustomJsonFormatter
    json:
        format: "'created', 'module', 'funcName', 'lineno', 'levelname', 'message'"

handlers:
    syslog:
        level: DEBUG
        class: logging.handlers.SysLogHandler
        facility: local0
        formatter: json
        address: [loghost, 514]

loggers:
    app_name:
        handlers: [syslog]
        propagate: True
        level: DEBUG

Next on to your loghost. I use my db server but any server with space for the logs will do. We need to configure it to accept remote requests and enable the local0 facility. I don't know why the local? are not enabled by default but no matter. Ubuntu's default syslogd is rsyslog and if that is what you got put the following in /etc/rsyslog.d/local0.conf

$ModLoad imudp
$UDPServerRun 514
$template msg,"{'%msg%\n"
local0.*        /var/log/local0;msg

Notice the "{" in the template. That should certainly not be there but for some reason rsyslog strips some of the message with just logging the %msg% like that. Why I have no idea and I wasn't going to go to the source to find out (syslog-ng did something similar so I am really at a loss). Now be sure to create the file and restart rsyslog.

sudo touch /var/log/local0
sudo chown syslog.adm /var/log/local0
sudo restart rsyslog
logger -p local0.info test test

Okay now its time to test. Logging messages from your app server should now be going to the central log host but you would be the luckiest person alive if that all worked without a hitch. If it doesn't be sure to check firewall rules, security groups, stuff like that.

Download the source.

0 Comments

Be the first to comment.

Django requests, logging and Lean Startups

Views: 4,980 Comments: 0

Tags:

Posted January 15, 2012 by derrick

So what does logging and django requests have anything to do with being a Lean Startup. I believe a lot. The model of the Lean Startup is to Build-Measure-Learn and you can only measure if you have the data for it. That is where logging comes into play. In this post I will show you how to integrate logging and django request objects.

I wrote a decorator to make the request available to logging by using a LoggerAdpater and RequestInfo make the request object dict-like.

class RequestInfo(object):

    def __init__(self, request):
        self.request = request

    def __getitem__(self, name):

        if name == 'request.host':
            return socket.gethostname()

        if name.startswith('request.meta.'):
            val = name.split('.')[2]
            try:
                return self.request.META[val.upper()]
            except KeyError as e:
                return None
        return eval('self.%s' % (name))

    def _get_attrs(self, obj):
        attrs = []
        for attr in dir(obj):
            try:
                if not attr.startswith('_') and \
                        not callable(getattr(obj, attr)):
                    attrs.append(attr)
            except AttributeError:
                pass
        return attrs

    def __iter__(self):
        keys = ['request.host']
        keys.extend(['request.%s' % (a) for a in
                self._get_attrs(self.request)])
        keys.extend(['request.session.%s' % (a) for a in
            self._get_attrs(self.request.session)])
        keys.extend(['request.user.%s' % (a) for a in
            self._get_attrs(self.request.user)])
        keys.extend(['request.meta.%s' % (a.lower()) for a in
            self.request.META.keys()])
        return keys.__iter__()

def logger(name):
    def wrap(func):
        def caller(*args, **kwargs):
            request = None
            for arg in args:
                if isinstance(arg, HttpRequest):
                    request = arg
            if 'logger' not in kwargs:
                if request is not None:
                    kwargs['logger'] = logging.LoggerAdapter(
                            logging.getLogger(name), RequestInfo(request))
                else:
                    kwargs['logger'] = logging.getLogger(name)
            return func(*args, **kwargs)
        return caller
    return wrap

its meant to be used with the views like this

@log.logger(__name__)
def home(request, logger=None):
    logger.info('user performed some action')

update your logging.yaml to use the new format. For more info on that please see my previous post.

request-json:
    (): log.CustomJsonFormatter
    format: "'created', 'request.host', 'module', 'funcName', 'lineno', 'levelname', 'request.meta.remote_addr',  'request.meta.http_user_agent', 'request.user.username', 'request.path', 'message'"

and the resulting log message should look something like this

{"host": "fastcgi22", "request.meta.http_user_agent": "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.15 (KHTML, like Gecko) Ubuntu/11.04 Chromium/18.0.996.0 Chrome/18.0.996.0 Safari/535.15", "request.path": "/", "created": 1326706577.275584, "module": "log", "funcName": "caller", "request.user.username": "derrick", "lineno": 114, "request.meta.remote_addr": "76.93.216.138", "message": "user performed some action", "levelname": "INFO"}

Now django requests are fully integrated with logging. Happy Trails.

Download the source.

Django logging, json, and syslog

Views: 97 Comments: 0

Tags:

Posted January 12, 2012 by derrick

In this post I will show you how to configure logging so your application servers can send info in nicely parseable json format to a centralized server for analysis.

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse',
        }
    },
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        }
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
    }
}

with this

import yaml
SITE_ROOT = os.path.dirname(os.path.realpath(__file__))
LOGGING = yaml.load(open(SITE_ROOT + '/logging.yaml', 'r'))

and put this into logging.yaml

version: 1
disable_existing_loggers: True
        
filters:
    require_debug_false:
        (): django.utils.log.RequireDebugFalse

handlers: 
    mail_admins:
        level: ERROR
        filters: [require_debug_false]
        class: django.utils.log.AdminEmailHandler

loggers:
    django.request:
        handlers: [mail_admins]
        propagate: True
        level: ERROR

class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def parse(self):
        return eval(self._fmt)

That allows for the format to be defined as python string. Append the json formatter, syslog handler and the app logger to their respective locations in logging.yaml.

formatters:
    (): log.CustomJsonFormatter
    json:
        format: "'created', 'module', 'funcName', 'lineno', 'levelname', 'message'"

handlers:
    syslog:
        level: DEBUG
        class: logging.handlers.SysLogHandler
        facility: local0
        formatter: json
        address: [loghost, 514]

loggers:
    app_name:
        handlers: [syslog]
        propagate: True
        level: DEBUG

$ModLoad imudp
$UDPServerRun 514
$template msg,"{'%msg%\n"
local0.*        /var/log/local0;msg

sudo touch /var/log/local0
sudo chown syslog.adm /var/log/local0
sudo restart rsyslog
logger -p local0.info test test

Download the source.

Adding http gzip support in Python

Views: 1,571 Comments: 1

Tags:

Posted May 11, 2011 by derrick

I have been doing a lot of http retrieval lately and the most efficient way to do that is with gzip compression enabled. Fortunately Python makes that really easy. All you have to do is derive from urllib2.HTTPHandler and override http_open().

import httplib, urllib, urllib2
class GzipHandler(urllib2.HTTPHandler):
    def http_open(self, req):
        req.add_header('Accept-encoding', 'gzip')
        r = self.do_open(httplib.HTTPConnection, req)
        if 'Content-Encoding'in r.headers and \
                r.headers['Content-Encoding'] == 'gzip':
            fp = gzip.GzipFile(fileobj=StringIO(r.read()))
        else:
            fp = r
        resp = urllib.addinfourl(fp, r.headers, r.url, r.code)
        resp.msg = r.msg
        return resp

The Accept-encoding header tells the server that this client supports gzip compression and if the Content-Encoding header is set to gzip the server returned an compressed response. Now you just need to build your opener.

def retrieve(url):
    request = urllib2.Request(url)
    opener = urllib2.build_opener(GzipHandler)
    return opener.open(request)

For more information see:

Download the source.

Searching Twitter with Python

Views: 1,641 Comments: 0

Tags:

Posted May 4, 2011 by derrick

I wanted to search Twitter and after reading their API docs I knew this was going to be a fun task given their native json support.

http://apiwiki.twitter.com/w/page/22554756/Twitter-Search-API-Method:-search

That means I would just have to handle pagination and creating the search query. Below is the resulting code.

twitter-search.py

#!/usr/bin/python2.7

import argparse
import datetime
import json
import urllib
import urlparse

class Twitter(object):

    search_url = 'http://search.twitter.com/search.json'

    def __init__(self, verbose=False):
        self.verbose = verbose
        super(Twitter, self).__init__()

    def search(self, query, until=None, rpp=100, max_results=None):

        results = []
        params = {
            'q': query,
            'rpp': rpp,
        }
        if until:
            params['until'] = until.strftime('%Y-%m-%d')

        if self.verbose:
            print(params)

        url = '%s?%s' % (self.search_url, urllib.urlencode(params))
        response = json.loads(urllib.urlopen(url).read())
        results.extend(response['results'])

        if len(results) >= max_results:
            return results
       
        while 'next_page' in response:
            url = self.search_url + response['next_page']
            response = json.loads(urllib.urlopen(url).read())
       
            if self.verbose:
                print('%s: %s' % (url, len(response['results'])))

            results.extend(response['results'])
            if len(results) >= max_results:
                break
        return results

    def search_last_day(self, *args, **kwargs):
        kwargs['until'] = datetime.datetime.now() - datetime.timedelta(days=1)
        return self.search(*args, **kwargs)

if __name__ == '__main__':

    parser = argparse.ArgumentParser(description='Search twitter')
    parser.add_argument('search', nargs=1)
    parser.add_argument('--rpp', dest='rpp', type=int, default=100, help='Results per page')
    parser.add_argument('-m', '--max-results', dest='max_results', type=int, default=100, help='Max results returned')
    parser.add_argument('-p', '--print-results', dest='print_results', action='store_true', help='Print the results')
    parser.add_argument('-v', '--verbose', dest='verbose', action='store_true', help='Turn verbose on')
    args = parser.parse_args()

    twitter = Twitter(verbose=args.verbose)
    results = twitter.search_last_day(args.search, rpp=args.rpp, max_results=args.max_results)
    print('Found %s items' % (len(results)))
    if args.verbose:
        json.dumps(results, indent=4)
    if args.print_results:
        for result in results:
            print('%s' % (result['text']))

Download the source.

Generating external ids with django

Views: 484 Comments: 0

Tags:

django

Posted September 21, 2010 by derrick

Sometimes you want an alternative to the traditional consecutive database ids i.e. to keep people from walking your site. Below is the code to generate an external id (a random 5 character field) when a model instance is created. I have used this on a couple of sites now and I am pretty happy with it.

models.py

from django.contrib.auth.models import User
from django.contrib.contenttypes import generic
from django.contrib.contenttypes.models import ContentType
from django.db import models

import random

class YourModelManager(models.Manager):

    @staticmethod
    def _random_id(prefix='', length=5):
        alphabet = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
        for x in range(length):
            prefix += random.choice(alphabet)
        return prefix

    @classmethod
    def _generate_external_id(cls):
        external_id = cls._random_id()
        while (YourModel.objects.filter(external_id=external_id).count() > 0):
            external_id = cls._random_id()
        return external_id

    def create(self, **kwargs):
        kwargs['external_id'] = self._generate_external_id()
        return self.get_query_set().create(**kwargs)

    def get_or_create(self, **kwargs):
        try:
            return self.get_query_set().get(**kwargs), False
        except self.model.DoesNotExist:
            kwargs['external_id'] = self._generate_external_id()
            return self.get_query_set().get_or_create(**kwargs)

class YourModel(models.Model):
    external_id = models.CharField(max_length=5, unique=True)
    objects = YourModelManager()

Download the source.

Capturing the output from ffmpeg

Views: 2,372 Comments: 0

Tags:

Posted August 20, 2010 by derrick

I needed to collect the output from ffmpeg for some profiling. It proved more challenging than I anticipated as ffmpeg writes the data unflushed to stderr making it unreadable using stdio. To get the data the stderr file descriptor has to set to NONBLOCK using fcntl. Here is the resulting Python code.

sample.py

def encode(filename, callback=None):
    cmd = 'ffmpeg -i "%s" -acodec libfaac -ab 128kb ' + \
          '-vcodec mpeg4 -b 1200kb -mbd 2 -flags +4mv ' + \
          '-trellis 2 -cmp 2 -subcmp 2 -s 320x180 "%s.mp4"'
    pipe = subprocess.Popen(
        shlex.split(cmd % (filename, os.path.splitext(filename)[0])),
        stderr=subprocess.PIPE,
        close_fds=True
    )
    fcntl.fcntl(
        pipe.stderr.fileno(),
        fcntl.F_SETFL,
        fcntl.fcntl(pipe.stderr.fileno(), fcntl.F_GETFL) | os.O_NONBLOCK,
    )
    # frame=   29 fps=  0 q=2.6 size=     114kB time=0.79 bitrate=1181.0kbits/s
    reo = re.compile("""\S+\s+(?P<frame>d+)  # frame
                         \s\S+\s+(?P<fps>\d+)           # fps
                         \sq=(?P<q>\S+)                    # q
                         \s\S+\s+(?P<size>\S+)          # size
                         \stime=(?P<time>\S+)           # time
                         \sbitrate=(?P<bitrate>[\d\.]+) # bitrate
                         """, re.X)
    while True:
        readx = select.select([pipe.stderr.fileno()], [], [])[0]
        if readx:
            chunk = pipe.stderr.read()
            if chunk == '':
                break
            m = reo.match(chunk)
            if m and callback:
                callback(m.groupdict())
        time.sleep(.1)

The complete script is located here.

Download the source.

Learning Erlang

Views: 161 Comments: 0

Tags:

erlang

Posted August 2, 2010 by derrick

I have been really loving Python lately but after reading this post. I thought it would be a good idea to check out Erlang. I have heard its concurrency and network support is out of this world and being in a knowledge based industry extra knowledge never hurts. This simple code snippet shows how much Erlang differs from the traditional procedural languages.

average(X) -> sum(X) / len(X).
sum([H|T]) -> H + sum(T);
sum([]) -> 0.
len([_|T]) -> 1 + len(T);
len([]) -> 0.

average takes at list X who calls sum and len. Both of those are recursive functions that split the list into the first element H and the remainder T. Variables must start with a capital letter and the '_' denotes the result is not used. Notice in this example no temporary variables were used. Talk about putting the "f" in functional. I can't wait to get to the concurrent stuff.

Download the source.

Upload files to S3 with progressbar

Views: 1,264 Comments: 0

Tags:

Posted July 28, 2010 by derrick

Here is a script I wrote to upload files to S3. It differs from s3put in that there is a progress indicator showing the ETA and the percentage uploaded and a summary after the transfer is complete.

s3upload.py

#!/usr/bin/python

import os
import sys
import optparse
import progressbar
import time

from boto.s3.connection import S3Connection
from boto.exception import S3ResponseError
from boto.s3.key import Key

AWS_ACCESS_KEY_ID = ''
AWS_SECRET_ACCESS_KEY = ''

pbar = None

def sizeof_fmt(num):
    for x in ['bytes','KB','MB','GB','TB']:
        if num < 1024.0:
            return "%3.1f%s" % (num, x)
        num /= 1024.0

def progress_callback(current, total):
    try:
        pbar.update(current)
    except AssertionError, e:
        print e

def upload_file(filename, bucket, prefix=None, reduced_redundancy=False):

    global pbar

    key = Key(bucket)
    if prefix:
        key.key = '%s/%s' % (prefix, filename)
    else:
        key.key = '%s' % (filename)

    size = os.stat(filename).st_size
    if size == 0:
        print 'Bad filesize for "%s"' % (filename)
        return 0

    widgets = [
        unicode(filename, errors='ignore').encode('utf-8'), ' ',
        progressbar.FileTransferSpeed(),
        ' <<<', progressbar.Bar(), '>>> ',
        progressbar.Percentage(), ' ', progressbar.ETA()
    ]
    pbar = progressbar.ProgressBar(widgets=widgets, maxval=size)
    pbar.start()

    try:
        key.set_contents_from_filename(
            filename,
            cb=progress_callback,
            num_cb=100,
            reduced_redundancy=reduced_redundancy,
        )
        key.set_acl('public-read')
    except IOError, e:
        print e
        return 0

    pbar.finish()
    return size

if __name__ == '__main__':

    parser = optparse.OptionParser(usage='usage: %prog [options] ')
    parser.add_option('-p', '--prefix', dest='prefix')
    parser.add_option('-r', '--reduced_rendundancy', dest='reduced_redundancy', action='store_true', default=False)
    (options, args) = parser.parse_args()

    if len(args) < 2:
        parser.print_help()
        sys.exit(1)

    conn = S3Connection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
    try:
        bucket = conn.get_bucket(args[0])
    except S3ResponseError, e:
        if e.error_code == 'NoSuchBucket':
            bucket = conn.create_bucket(args[0])
        else:
            raise e

    stime = time.time()
    total_bytes = 0
    count = 0
    for arg in args[1:]:
        size = upload_file(arg, bucket, options.prefix, options.reduced_redundancy)
        total_bytes += size
        count += 1

    if len(args) > 2:
        print
        print '%s files %s at %.2f kb/s' % (count, sizeof_fmt(total_bytes), (total_bytes / 1024)/time.time() - stime))

To install it run these commands

sudo pip install boto progressbar
git clone git://gist.github.com/510222.git s3upload-gist
cd s3upload-gist
vim s3upload # Add your AWS settings
chmod 755 s3upload-gist/s3upload
sudo mv s3upload-gist/s3upload /usr/local/bin
rm -rf s3upload-gist

Download the source.

Django Facebook Authentication Backend

Views: 1,124 Comments: 1

Tags:

Posted July 15, 2010 by derrick

Here is a Django authentication backend I wrote using Facebook's amazingly simple Graph API. It logs the user in using their Facebook credentials so you site doesn't have to worry about creating user profiles, validating, etc. See

Define the facebook tokens in settings.py and replace with the name of your app. You will probably want to modify the scope on the authorize link in the template, see the authentication permissions link.

backends.py

from django.conf import settings
from django.contrib.auth import models as auth_models

import cgi
import urllib
import simplejson

from <app_name> import models

class FacebookBackend:

    def authenticate(self, token=None):

        facebook_session = models.FacebookSession.objects.get(
            access_token=token,
        )

        profile = facebook_session.query('me')

        try:
            user = auth_models.User.objects.get(username=profile['id'])
        except auth_models.User.DoesNotExist, e:
            user = auth_models.User(username=profile['id'])

        user.set_unusable_password()
        user.email = profile['email']
        user.first_name = profile['first_name']
        user.last_name = profile['last_name']
        user.save()

        try:
            models.FacebookSession.objects.get(uid=profile['id']).delete()
        except models.FacebookSession.DoesNotExist, e:
            pass

        facebook_session.uid = profile['id']
        facebook_session.user = user
        facebook_session.save()

        return user

    def get_user(self, user_id):

        try:
            return auth_models.User.objects.get(pk=user_id)
        except auth_models.User.DoesNotExist:
            return None

models.py

from django.db import models
from django.contrib.auth.models import User

class FacebookSessionError(Exception):
    def __init__(self, error_type, message):
        self.message = message
        self.type = error_type
    def get_message(self):
        return self.message
    def get_type(self):
        return self.type
    def __unicode__(self):
        return u'%s: "%s"' % (self.type, self.message)

class FacebookSession(models.Model):

    access_token = models.CharField(max_length=103, unique=True)
    expires = models.IntegerField(null=True)

    user = models.ForeignKey(User, null=True)
    uid = models.BigIntegerField(unique=True, null=True)

    class Meta:
        unique_together = (('user', 'uid'), ('access_token', 'expires'))

    def query(self, object_id, connection_type=None, metadata=False):
        import urllib
        import simplejson

        url = 'https://graph.facebook.com/%s' % (object_id)
        if connection_type:
            url += '/%s' % (connection_type)

        params = {'access_token': self.access_token}
        if metadata:
            params['metadata'] = 1

        url += '?' + urllib.urlencode(params)
        response = simplejson.load(urllib.urlopen(url))
        if 'error' in response:
            error = response['error']
            raise FacebookSessionError(error['type'], error['message'])
        return response

views.py

from django.contrib import auth
from django.http import HttpResponseRedirect
from django.shortcuts import render_to_response
from django.template import RequestContext

import cgi
import simplejson
import urllib

from <app_name> import settings

def login(request):
    error = None

    if request.user.is_authenticated():
        return HttpResponseRedirect('/yay/')

    if request.GET:
        if 'code' in request.GET:
            args = {
                'client_id': settings.FACEBOOK_APP_ID,
                'redirect_uri': settings.FACEBOOK_REDIRECT_URI,
                'client_secret': settings.FACEBOOK_API_SECRET,
                'code': request.GET['code'],
            }

            url = 'https://graph.facebook.com/oauth/access_token?' + \
                    urllib.urlencode(args)
            response = cgi.parse_qs(urllib.urlopen(url).read())
            access_token = response['access_token'][0]
            expires = response['expires'][0]

            facebook_session = models.FacebookSession.objects.get_or_create(
                access_token=access_token,
            )[0]

            facebook_session.expires = expires
            facebook_session.save()

            user = auth.authenticate(token=access_token)
            if user:
                if user.is_active:
                    auth.login(request, user)
                    return HttpResponseRedirect('/yay/')
                else:
                    error = 'AUTH_DISABLED'
            else:
                error = 'AUTH_FAILED'
        elif 'error_reason' in request.GET:
            error = 'AUTH_DENIED'

    template_context = {'settings': settings, 'error': error}
    return render_to_response('login.html', template_context, context_instance=RequestContext(request))

settings.py

FACEBOOK_APP_ID = ''
FACEBOOK_API_KEY = ''
FACEBOOK_API_SECRET = ''
FACEBOOK_REDIRECT_URI = 'http://example.com/login/'

AUTHENTICATION_BACKENDS = (
    '<app_name>.backends.FacebookBackend',
)

login.html

{% if error %}
      {% if error == 'AUTH_FAILED' %}
          <p>Authentication failed</p>
      {% else %}{% if error == 'AUTH_DISABLED' %}
          <p>Your account is disabled</p>
      {% else %}{% if error == 'AUTH_DENIED' %}
          <p>You did not allow access</p>
       {% endif %}{% endif %}{% endif %}
  {% else %}
    <a href="https://graph.facebook.com/oauth/authorize?client_id={{ settings.FACEBOOK_APP_ID }}&redirect_uri={{ settings.FACEBOOK_REDIRECT_URI }}&scope=publish_stream,email&display=popup">
      <img src="http://developers.facebook.com/images/devsite/login-button.png"/>
    </a>
  {% endif %}

Download the source.

Simple python syslog wrapper class

Views: 421 Comments: 1

Tags:

Posted July 15, 2010 by derrick

I just wanted a simple wrapper around syslog. The Python logging module is good but it was too heavyweight for what I needed. Here is simple logging class for syslog. It has an optional decorator to provide the function name to syslog which I find useful for debugging.

syslogging.py

#!/usr/bin/python

import syslog

class SysLogging:
    def __init__(self, facility, prefix=None):
        self.facility = facility
        if prefix:
            syslog.openlog(prefix)
    def _logit(self, priority, message):
        syslog.syslog(self.facility | priority, '%s' % (message))
    def debug(self, message):
        self._logit(syslog.LOG_DEBUG, message)
    def info(self, message):
        self._logit(syslog.LOG_INFO, message)
    def notice(self, message):
        self._logit(syslog.LOG_NOTICE, message)
    def warning(self, message):
        self._logit(syslog.LOG_WARNING, message)
    def error(self, message):
        self._logit(syslog.LOG_ERR, message)
    def crit(self, message):
        self._logit(syslog.LOG_CRIT, message)
    def alert(self, message):
        self._logit(syslog.LOG_ALERT, message)
    def emerg(self, message):
        self._logit(syslog.LOG_EMERG, message)

def syslogging(func):
    def caller(*args, **kwargs):
        if 'logger' not in kwargs:
            kwargs['logger'] = SysLogging(syslog.LOG_LOCAL2, func.__name__)
        return func(*args, **kwargs)
    return caller

@syslogging
def test_func(arg1, arg2=None, logger=None):
    logger.info('%s %s' % (arg1, arg2))

if __name__ == '__main__':
    test_func(1, 'two')

Download the source.

Python function runtime logging decorator

Views: 798 Comments: 0

Tags:

Posted July 15, 2010 by derrick

When profiling it can be useful to log the amount of time that is spent in a function. With Python that is super easy to do with decorators.

logtime.py

#!/usr/bin/python

import time
import syslog

def logtime(func):
    def caller(*args, **kwargs):
        stime = time.time()
        ret = func(*args, **kwargs)
        syslog.syslog(
            syslog.LOG_LOCAL2 | syslog.LOG_INFO,
            '%s=%s\n' % (func.__name__, time.time() - stime))
        return ret
    return caller

@logtime
def test_func(arg1, arg2=None):
    print arg1, arg2
    time.sleep(1)

if __name__ == '__main__':
    test_func(1, 2)

logtime will log the time spent in the function to syslog.

Jul 14 15:05:01 olomai python: test_func=1.00114893913

Download the source.

Comparison of IN, GROUP BY and COUNT using Hibernate, Django and SQLAlchemy

Views: 1,244 Comments: 0

Tags:

Posted July 7, 2010 by derrick

The other day I wrote about how to do a IN and GROUP BY query using Java's de facto ORM, Hibernate. I thought it would be interesting to see how other ORMs handled the same query. This is the query I want to generate:

SELECT COUNT(*),state FROM download_request WHERE id IN (<id list>) GROUP BY state;

Below is the code, output and SQL generated for the three ORMs.

Hibernate

class HibernateDAO implements ApplicationDAO {
public Map getStateCounts(final Collection ids) {
  HibernateSession hibernateSession = new HibernateSession();
  Session session = hibernateSession.getSession();
  Criteria criteria = session.createCriteria(DownloadRequestEntity.class)
	.add(Restrictions.in("id", ids));
  ProjectionList projectionList = Projections.projectionList();
  projectionList.add(Projections.groupProperty("state"));
  projectionList.add(Projections.rowCount());
  criteria.setProjection(projectionList);
  List results = criteria.list();
  Map stateMap = new HashMap();
  for(Object[] obj: results) {
      	DownloadState downloadState = (DownloadState)obj[0];
       	stateMap.put(downloadState.getDescription().toLowerCase(), (Integer)obj[1]);
  }
  hibernateSession.closeSession();
  return stateMap;
}
public static void main(String args[]) {
    HibernateDAO downloadRequestDAO = new HibernateDAO();	
    Collection ids = new ArrayList();
    for (int i = 1000;  i < 1010; i++ )
        ids.add(i);
    Map stateCounts =  downloadRequestDAO.getStateCounts(ids);
    for (String state: stateCounts.keySet()) {
        System.out.println(state + ": " + stateCounts.get(state));
    }
}
}

Output

failed: 5
downloaded: 1
completed: 4

SQL

select this_.state as y0_, count(*) as y1_ from download_request this_ 
where this_.id in (1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009) 
group by this_.state

Django

counts = models.DownloadRequest.objects.filter(
    id__in=range(1000, 1010),
).values('state').annotate(Count('state'))
for count in counts:
    print count

Output

{'state': u'FAILED', 'state__count': 5}
{'state': u'COMPLETED', 'state__count': 4}
{'state': u'DOWNLOADED', 'state__count': 1}

SQL

SELECT `download_request`.`state`, COUNT(`download_request`.`state`) 
AS `state__count` FROM `download_request` 
WHERE `download_request`.`id` IN (1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009) 
GROUP BY `download_request`.`state` ORDER BY NULL

SQLAlchmey

query = session.query(
    func.count(DownloadRequest.state), DownloadRequest.state,
).filter(
    DownloadRequest.id.in_(range(1000,1010)),
).group_by(DownloadRequest.state)
for count in query.all():
    print count

Output

(4L, 'COMPLETED')
(1L, 'DOWNLOADED')
(5L, 'FAILED')

SQL

SELECT count(download_request.state) AS count_1, download_request.state 
AS download_request_state FROM download_request 
WHERE download_request.id IN (1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009) 
GROUP BY download_request.state

As you can see SQLAlchemy is the most similar to SQL, django's is the briefest and Hibernate (obviously) is the most Java-like. Of the three I'd say I like SQLAlchemy the best as it is the most similar to SQL and me being from an SQL background it is the most natural. However all three get the job done and it is always great to have options.

Download the source.

IN, GROUP BY and COUNT with Hibernate

Views: 3,567 Comments: 0

Tags:

hibernate

Posted July 2, 2010 by derrick

I needed to make the following SQL query with Hibernate

SELECT COUNT(*), state FROM download_request WHERE id IN (<id list>) GROUP BY state;

and being new to Hibernate it came out a lot differently than how I thought it would. To perform the IN query a Criteria query needs to be created

Criteria criteria = session.createCriteria(DownloadRequestEntity.class)
    .add(Restrictions.in("id", ids))

For the count and order by a Projection needs to be added to the criteria

ProjectionList projectionList = Projections.projectionList();
projectionList.add(Projections.groupProperty("state"));
projectionList.add(Projections.rowCount());
criteria.setProjection(projectionList);

This is the resulting code

public Map getStateCounts(final Collection ids) {
  HibernateSession hibernateSession = new HibernateSession();
  Session session = hibernateSession.getSession();
  Criteria criteria = session.createCriteria(DownloadRequestEntity.class)
	.add(Restrictions.in("id", ids));
  ProjectionList projectionList = Projections.projectionList();
  projectionList.add(Projections.groupProperty("state"));
  projectionList.add(Projections.rowCount());
  criteria.setProjection(projectionList);
  List results = criteria.list();
  Map stateMap = new HashMap();
  for(Object[] obj: results) {
      	DownloadState downloadState = (DownloadState)obj[0];
       	stateMap.put(downloadState.getDescription().toLowerCase(), (Integer)obj[1]);
  }
  hibernateSession.closeSession();
  return stateMap;
}

Something completely different from what I expected. That's what I love about solving problems sometimes the solution is something you might never expect.

Download the source.

Init script for fastcgi and php on Ubuntu

Views: 614 Comments: 0

Tags:

Posted July 2, 2010 by derrick

This is an init script to run spawn-fcgi and php on Ubuntu. Its adapted from Aaron Schaefer's excellent post on how to run wordpress on nginx - the configuration this site runs on.

fastcgi-php

#!/bin/sh

## BEGIN INIT INFO
# Provides: FastCGI servers for PHP
# Required-Start: networking
# Required-Stop: networking
# Default-Start: 2 3 4 5
# Default-Stop: S 0 1 6
# Short-Description: Start FastCGI servers with PHP.
# Description: Start PHP with spawn-fcgi. For use with nginx and lighttpd.
#
#
### END INIT INFO
#
# Author: Derrick Petzold
#
# http://derrickpetzold.com/index.php/2010/07/02/init-script-fastcgi-php-ubuntu/
#

RUN_USER=nobody
RUN_GROUP=nogroup

LISTEN_ADDRESS=127.0.0.1
LISTEN_PORT=53217

SPAWN_FCGI=/opt/lighttpd/bin/spawn-fcgi
PHP_CGI=/opt/php5/bin/php-cgi
PID_FILE=/var/run/fastcgi-php.pid

d_start() {
    if [ -f $PID_FILE ]; then
      echo -n " already running"
    else
        start-stop-daemon --start -p $PID_FILE \
            --exec /usr/bin/env -- $SPAWN_FCGI -f $PHP_CGI \
            -u $RUN_USER -g $RUN_GROUP -a $LISTEN_ADDRESS -p $LISTEN_PORT \
            -P $PID_FILE
    fi
}

d_stop() {
    start-stop-daemon --stop --quiet --pidfile $PID_FILE \
                      || echo -n " not running"
    if [ -f $PID_FILE ]; then
        rm $PID_FILE
    fi
}

case "$1" in
  start)
    echo -n "Starting FastCGI: $0 "
    d_start
    echo "."
    ;;

  stop)
    echo -n "Stopping FastCGI: $0 "
    d_stop
    echo "."
    ;;
  restart)
    echo -n "Restarting FastCGI: $0 "
    d_stop
    sleep 1
    d_start
    ;;
  *)
    echo "usage: $0 {start|stop|restart}"
    ;;
esac

To install it follow the instructions below

git clone git://gist.github.com/510245.git fastcgi-php.gist
vim fastcgi-php.gist/fastcgi-php # Update with your pathnames
chmod 755 fastcgi-php.gist/fastcgi-php
mv fastcgi-php.gist/fastcgi-php /etc/init.d/
update-rc.d fastcgi-php defaults
/etc/init.d/fastcgi-php start
rm -rf fastcgi-php.gist

Download the source.

Recent Articles

More >>

Twitter

Big day today. Released first django app https://t.co/TJryrhcc. Its what powers the comments on my blog. Try them out https://t.co/VuRrxL55 1 month, 1 week ago
The django source tree increased by 300m with the addition of localflavor. That's going to be murder on svn... 1 month, 2 weeks ago
In regards to my News Years Resolution: I have already written no less than 2 shell scripts today. So, so much for that. 1 month, 2 weeks ago
@obeattie I think its really great. Tho I twitted after the eval :). 1 month, 2 weeks ago
Another nginx patched submitted - http://t.co/hO3LAOW5. Adds support for uwsgi_host and uwsgi_port. Let's see how it goes. 1 month, 2 weeks ago

Blog Roll

derrickpetzold.com

My home on the web. Uses the Django web framework, uwsgi as the WSGI server, nginx as the media server and load balancer, pygments for the syntax highlighting. Author and designer.

crowdtube.tv

CrowdTube.tv streams trending videos to your browser like a never ending tv show. This was Cory Shaw's awesome idea. I was responsible for the backend development.

dmusic.bz

dMusic.bz is Pandora clone written using Django and JavaScript. It uses last.fm to find the music relationships. Author and designer.

ilovephotos.com

ilovephotos.com is a photo sharing and tagging website. Facial detection was run on the photos so the bounding boxes were already drawn around the faces eliminating a step in a tedious process. I authored views and the process to run facial detection on the photos and upload them to S3.

kindfish.com

Kindfish.com was BlueLava's first photo site. It was event based and slideshows would be created from the albums. I authored the views and process to generate the slideshows from the photos on EC2.

Below is a snippet of my resume. Click here to view the full version in pdf, its proper format.

0 Comments

Login to comment:

twitter-search.py

models.py

sample.py

s3upload.py

backends.py

models.py

views.py

settings.py

login.html

syslogging.py

logtime.py

Hibernate

Output

SQL

Django

Output

SQL

SQLAlchmey

Output

SQL

fastcgi-php

Tag Cloud

Recent Articles

Subscribe

Twitter

Blog Roll

dmusic.bz