Procs¶
This is a auto-generated list of supported Proc
classes.
Celery¶
-
class
hirefire.procs.celery.
CeleryProc
(name=None, queues=['celery'], app=None)[source]¶ A proc class for the Celery library.
Parameters: Declarative example:
from celery import Celery from hirefire.procs.celery import CeleryProc celery = Celery('myproject', broker='amqp://guest@localhost//') class WorkerProc(CeleryProc): name = 'worker' queues = ['celery'] app = celery
Or a simpler variant:
worker_proc = CeleryProc('worker', queues=['celery'], app=celery)
In case you use one of the non-standard Celery clients (e.g. django-celery) you can leave the
app
attribute empty because Celery will automatically find the correct Celery app:from hirefire.procs.celery import CeleryProc class WorkerProc(CeleryProc): name = 'worker' queues = ['celery']
Querying the tasks that are on the workers is a more expensive process, and if you’re sure that you don’t need them, then you can improve the response time by not looking for some statuses. The default statuses that are looked for are
active
,reserved
, andscheduled
. You can configure to not look for those by overriding theinspect_statuses
property. For example, this proc would not look at any tasks held by the workers.class WorkerProc(CeleryProc): name = 'worker' queues = ['celery'] inspect_statuses = []
scheduled
tasks are tasks that have been triggered with aneta
, the most common example of which is usingretry
on tasks. If you’re sure you aren’t using these tasks, you can skip querying for these tasks.reserved
tasks are tasks that have been taken from the queue by the main process (coordinator) on the worker dyno, but have not yet been given to a worker run. If you’ve configured Celery to only fetch the tasks that it is currently running, then you may be able to skip querying for these tasks. See http://docs.celeryproject.org/en/latest/userguide/optimizing.html#prefetch-limits form more information.active
tasks are currently running tasks. If your tasks are short-lived enough, then you may not need to look for these tasks. If you choose to not look at active tasks, look out forWorkerLostError
exceptions. See https://github.com/celery/celery/issues/2839 for more information.If you have a particular simple case, you can use a shortcut to eliminate one inspect call when inspecting statuses. The
active_queues
inspect call is needed to mapexchange
androuting_key
back to the celeryqueue
that it is for. If all of yourqueue
,exchange
, androuting_key
are the same (which is the default in Celery), then you can use thesimple_queues = True
flag to note that all the queues in the proc use the same name for theirexchange
androuting_key
. This defaults toFalse
for backward compatibility, but if your queues are using this simple setup, you’re encouraged to use it like so:class WorkerProc(CeleryProc): name = 'worker' queues = ['celery'] simple_queues = True
Because of how this is implemented, you will almost certainly wish to use this feature on all of your procs, or on none of them. This is because both variants have separate caches that make separate calls to the inspect methods, so having both kinds present will mean that the inspect calls will be run twice.
-
app
= None¶ The Celery app to check for the queues (optional).
-
inspect_statuses
= ['active', 'reserved', 'scheduled']¶ The Celery task status to check for on workers (optional). Valid options are ‘active’, ‘reserved’, and ‘scheduled’.
-
name
= None¶ The name of the proc (required).
-
queues
= ['celery']¶ The list of queues to check (required).
-
simple_queues
= False¶ Whether or not the exchange and routing_key are the same as the queue name for the queues in this proc. Default: False.
-
HotQueue¶
-
class
hirefire.procs.hotqueue.
HotQueueProc
(name=None, queues=[], connection_params={})[source]¶ A proc class for the HotQueue library.
Parameters: Example:
from hirefire.procs.hotqueue import HotQueueProc class WorkerHotQueueProc(HotQueueProc): name = 'worker' queues = ['myqueue'] connection_params = { 'host': 'localhost', 'port': 6379, 'db': 0, }
-
client
(queue)[source]¶ Given one of the configured queues returns a
hotqueue.HotQueue
instance with theconnection_params
.
-
connection_params
= {}¶ The connection parameter to use by default (optional).
-
name
= None¶ The name of the proc (required).
-
queues
= []¶ The list of queues to check (required).
-
Huey¶
Queues¶
-
class
hirefire.procs.queues.
QueuesProc
(name=None, queues=[])[source]¶ A proc class for the queues library.
Parameters: - name (str) – the name of the proc (required)
- queues (str or list of str or
queues.queues.Queue
) – list of queue names to check (required)
Example:
from hirefire.procs.queues import QueuesProc class WorkerQueuesProc(QueuesProc): name = 'worker' queues = ['default', 'thumbnails']
-
name
= None¶ The name of the proc (required).
-
queues
= []¶ The list of queues to check (required).
RQ¶
-
class
hirefire.procs.rq.
RQProc
(name=None, queues=['default'], connection=None)[source]¶ A proc class for the RQ library.
Parameters: Example:
from hirefire.procs.rq import RQProc class WorkerRQProc(RQProc): name = 'worker' queues = ['high', 'default', 'low']
-
client
(queue)[source]¶ Given one of the configured queues returns a
rq.Queue
instance using theconnection
.
-
connection
= None¶ The connection to use for the queues (optional).
-
name
= None¶ The name of the proc (required).
-
queues
= ['default']¶ The list of queues to check (required).
-