HireFire

This is a Python package for HireFire – The Heroku Process Manager:

HireFire has the ability to automatically scale your web and worker dynos up and down when necessary. When new jobs are queued in to your application’s worker queue [..], HireFire will spin up new worker dynos to process these jobs. When the queue is empty, HireFire will shut down the worker dynos again so you’re not paying for idle workers.

HireFire also has the ability to scale your web dynos. When your web application experiences heavy traffic during certain times of the day, or if you’ve been featured somewhere, chances are your application’s backlog might grow to a point that your web application will run dramatically slow, or even worse, it might result in a timeout. In order to prevent this, HireFire will automatically scale your web dynos up when traffic increases to ensure that your application runs fast at all times. When traffic decreases, HireFire will spin down your web dynos again.

—from the HireFire frontpage

It supports the following Python queuing systems as backends:

Feel free to contribute other backends if you’re using a different queuing system.

Note

This package is currently (02/16/2013) targetting the HireFire beta version (“manager.hirefire.io”) NOT the old product (“monitor.hirefireapp.com”).

See the beta support forum to get started with this improved version.

Installation

Install the HireFire package with your favorite installer, e.g.:

pip install HireFire

Sign up for the HireFire (beta) and set the HIREFIRE_TOKEN environment variable with the Heroku CLI as provided on the specific HireFire application page, e.g.:

heroku config:set HIREFIRE_TOKEN=f69f0c0ddebe041248daf187caa6abb3e5d943ca

Now follow the quickstart guide below and don’t forget to tweak the options in the HireFire management system.

Configuration

The hirefire Python package currently supports only one framework: Django. Implementations for other frameworks are planned but haven’t been worked on: Flask, Pyramid (PasteDeploy), WSGI middleware, ..

Feel free to contribute one if you can’t wait.

Django

Setting up HireFire support for Django is easy:

  1. Add 'hirefire.contrib.django.middleware.HireFireMiddleware' to your ``MIDDLEWARE_CLASSES setting:

    MIDDLEWARE_CLASSES = [
        'hirefire.contrib.django.middleware.HireFireMiddleware',
        # ...
    ]
    

    Make sure it’s the first item in the list/tuple.

  2. Define as many hirefire.Proc subclasses as you want HireFire to monitor. Have a look at your Procfile file to do it.

    For example here is a Procfile with the following content which uses RQ for the worker proccess:

    web: python manage.py runserver
    worker: DJANGO_SETTINGS_MODULE=mysite.settings rqworker high default low

    Define a RQProc subclass somewhere in your Django project, e.g. mysite/procs.py, with the appropriate attributes (name and queues):

    from hirefire.procs.rq import RQProc
    
    class WorkerProc(RQProc):
        name = 'worker'
        queues = ['high', 'default', 'low']
    

    See the procs API documentation if you’re using another backend.

  3. Set the HIREFIRE_PROCS setting to a list of dotted paths to your procs. For the above example proc:

    HIREFIRE_PROCS = ['mysite.procs.WorkerProc']
    
  4. Set the HIREFIRE_TOKEN setting to the token that HireFire show on the specific application page (optional):

    HIREFIRE_TOKEN = 'f69f0c0ddebe041248daf187caa6abb3e5d943ca'
    

    This is only needed if you haven’t set the HIREFIRE_TOKEN environment variable already (see above).

  5. Check that the middleware has been correctly setup by opening the following URL in a browser:

    http://localhost:8000/hirefire/test

    You should see an empty page with ‘HireFire Middleware Found!’.

    You can also have a look at the page that HireFire checks to get the number of current tasks:

    http://localhost:8000/hirefire/<HIREFIRE_TOKEN>/info

    where <HIREFIRE_TOKEN> needs to be replaced with your token or – in case you haven’t set the token in your settings or environment – just use development.

Proc backends

Two base classes are includes that you can use to implement custom backends. All the other contributed backends use those base classes, too.

hirefire.procs.Proc

class hirefire.procs.Proc(name=None, queues=None)[source]

The base proc class. Use this to implement custom queues or other behaviours, e.g.:

import mysite.sekrit
from hirefire import procs

class MyCustomProc(procs.Proc):
    name = 'worker'
    queues = ['default']

    def quantity(self):
        return sum([mysite.sekrit.count(queue)
                    for queue in self.queues])
Parameters:
  • name (str) – the name of the proc (required)
  • queues (str or list of str) – list of queue names to check (required)
name = None

The name of the proc

quantity(*args, **kwargs)[source]

Returns the aggregated number of tasks of the proc queues.

Needs to be implemented in a subclass.

queues = []

The list of queues to check

hirefire.procs.ClientProc

class hirefire.procs.ClientProc(*args, **kwargs)[source]

A special subclass of the Proc class that instantiates a list of clients for each given queue, e.g.:

import mysite.sekrit
from hirefire import procs

class MyCustomProc(procs.ClientProc):
    name = 'worker'
    queues = ['default']

    def client(self, queue):
        return mysite.sekrit.Client(queue)

    def quantity(self):
        return sum([client.count(queue)
                    for client in self.clients])

See the implementation of the RQProc class for an example.

client(queue, *args, **kwargs)[source]

Returns a client instance for the given queue to be used in the quantity method.

Needs to be implemented in a subclass.

quantity(*args, **kwargs)

Returns the aggregated number of tasks of the proc queues.

Needs to be implemented in a subclass.

Contributed backends

See the following API overview of the other supported queuing backends.

Issues & Feedback

For bug reports, feature requests and general feedback, please use the Github issue tracker.

Thanks

Many thanks to the folks at Hirefire for building a great tool for the Heroku ecosystem.

Authors

Project Versions

Table Of Contents

Next topic

Celery

This Page