Procs

This is a auto-generated list of supported Proc classes.

Celery

class hirefire.procs.celery.CeleryProc(name=None, queues=['celery'], app=None)[source]

A proc class for the Celery library.

Parameters:
  • name (str) – the name of the proc (required)
  • queues (str or list) – list of queue names to check (required)
  • app (Celery) – the Celery app to check for the queues (optional)

Declarative example:

from celery import Celery
from hirefire.procs.celery import CeleryProc

celery = Celery('myproject', broker='amqp://guest@localhost//')

class WorkerProc(CeleryProc):
    name = 'worker'
    queues = ['celery']
    app = celery

Or a simpler variant:

worker_proc = CeleryProc('worker', queues=['celery'], app=celery)

In case you use one of the non-standard Celery clients (e.g. django-celery) you can leave the app attribute empty because Celery will automatically find the correct Celery app:

from hirefire.procs.celery import CeleryProc

class WorkerProc(CeleryProc):
    name = 'worker'
    queues = ['celery']

Querying the tasks that are on the workers is a more expensive process, and if you’re sure that you don’t need them, then you can improve the response time by not looking for some statuses. The default statuses that are looked for are active, reserved, and scheduled. You can configure to not look for those by overriding the inspect_statuses property. For example, this proc would not look at any tasks held by the workers.

class WorkerProc(CeleryProc):
    name = 'worker'
    queues = ['celery']
    inspect_statuses = []

scheduled tasks are tasks that have been triggered with an eta, the most common example of which is using retry on tasks. If you’re sure you aren’t using these tasks, you can skip querying for these tasks.

reserved tasks are tasks that have been taken from the queue by the main process (coordinator) on the worker dyno, but have not yet been given to a worker run. If you’ve configured Celery to only fetch the tasks that it is currently running, then you may be able to skip querying for these tasks. See http://docs.celeryproject.org/en/latest/userguide/optimizing.html#prefetch-limits form more information.

active tasks are currently running tasks. If your tasks are short-lived enough, then you may not need to look for these tasks. If you choose to not look at active tasks, look out for WorkerLostError exceptions. See https://github.com/celery/celery/issues/2839 for more information.

If you have a particular simple case, you can use a shortcut to eliminate one inspect call when inspecting statuses. The active_queues inspect call is needed to map exchange and routing_key back to the celery queue that it is for. If all of your queue, exchange, and routing_key are the same (which is the default in Celery), then you can use the simple_queues = True flag to note that all the queues in the proc use the same name for their exchange and routing_key. This defaults to False for backward compatibility, but if your queues are using this simple setup, you’re encouraged to use it like so:

class WorkerProc(CeleryProc):
    name = 'worker'
    queues = ['celery']
    simple_queues = True

Because of how this is implemented, you will almost certainly wish to use this feature on all of your procs, or on none of them. This is because both variants have separate caches that make separate calls to the inspect methods, so having both kinds present will mean that the inspect calls will be run twice.

app = None

The Celery app to check for the queues (optional).

inspect_count(cache)[source]

Use Celery’s inspect() methods to see tasks on workers.

inspect_statuses = ['active', 'reserved', 'scheduled']

The Celery task status to check for on workers (optional). Valid options are ‘active’, ‘reserved’, and ‘scheduled’.

name = None

The name of the proc (required).

quantity(cache=None, **kwargs)[source]

Returns the aggregated number of tasks of the proc queues.

queues = ['celery']

The list of queues to check (required).

simple_queues = False

Whether or not the exchange and routing_key are the same as the queue name for the queues in this proc. Default: False.

HotQueue

class hirefire.procs.hotqueue.HotQueueProc(name=None, queues=[], connection_params={})[source]

A proc class for the HotQueue library.

Parameters:
  • name (str) – the name of the proc (required)
  • queues (str or list) – list of queue names to check (required)
  • connection_params (dict) – the connection parameter to use by default (optional)

Example:

from hirefire.procs.hotqueue import HotQueueProc

class WorkerHotQueueProc(HotQueueProc):
    name = 'worker'
    queues = ['myqueue']
    connection_params = {
        'host': 'localhost',
        'port': 6379,
        'db': 0,
    }
client(queue)[source]

Given one of the configured queues returns a hotqueue.HotQueue instance with the connection_params.

connection_params = {}

The connection parameter to use by default (optional).

name = None

The name of the proc (required).

quantity(**kwargs)[source]

Returns the aggregated number of tasks of the proc queues.

queues = []

The list of queues to check (required).

Huey

Queues

class hirefire.procs.queues.QueuesProc(name=None, queues=[])[source]

A proc class for the queues library.

Parameters:
  • name (str) – the name of the proc (required)
  • queues (str or list of str or queues.queues.Queue) – list of queue names to check (required)

Example:

from hirefire.procs.queues import QueuesProc

class WorkerQueuesProc(QueuesProc):
    name = 'worker'
    queues = ['default', 'thumbnails']
client(queue)[source]

Given one of the configured queues returns a queues.queues.Queue instance.

name = None

The name of the proc (required).

quantity(**kwargs)[source]

Returns the aggregated number of tasks of the proc queues.

queues = []

The list of queues to check (required).

RQ

class hirefire.procs.rq.RQProc(name=None, queues=['default'], connection=None)[source]

A proc class for the RQ library.

Parameters:
  • name (str) – the name of the proc (required)
  • queues (str or list) – list of queue names to check (required)
  • connection (redis.Redis) – the connection to use for the queues (optional)

Example:

from hirefire.procs.rq import RQProc

class WorkerRQProc(RQProc):
    name = 'worker'
    queues = ['high', 'default', 'low']
client(queue)[source]

Given one of the configured queues returns a rq.Queue instance using the connection.

connection = None

The connection to use for the queues (optional).

name = None

The name of the proc (required).

quantity(**kwargs)[source]

Returns the aggregated number of tasks of the proc queues.

queues = ['default']

The list of queues to check (required).