Linked Tasks#

Sometimes a task lends itself to being performed in multiple parallel processes, or takes long enough that it should logically be broken down into several sequential tasks. Celery 3.0 added explicit support for linking tasks together in a few different ways, and django-user-tasks generally supports these. The ones which have been specifically tested so far are listed below.

Note that it’s usually best for all of the tasks in a grouping to inherit from UserTaskMixin if any of them do. Any tasks in the group which do not inherit from that class won’t be factored into the status of the grouping as a whole.

Chains#

A chain is a sequence of Celery tasks, where the completion of one task in the chain causes the next one to be sent to the broker for a worker process to start working on. django-user-tasks recognizes such chains and creates a separate status record for the chain as a whole if any of the tasks within it inherit from UserTaskMixin. Each task in the chain which has that mixin will update its own status record and propagate changes to the overall chain status record as appropriate.

The individual tasks in the chain will each have their parent field set to the status record for the whole chain, while the parent status will have the following characteristics:

  • name is set by providing a “user_task_name” keyword argument to at least one of the child tasks (if none of them do so, the name will be empty). The first name provided in this manner will be used.

  • is_container is True.

  • task_class is “celery.chain”.

  • total_steps and completed_steps are kept up to date as the sum of those fields in all the child tasks.

  • state is “In Progress” while any of the child tasks are being executed or retried (unless one of them uses UserTaskStatus.set_state() to set a custom state, in which case that will propagate to the parent as well).

A failure in one of the child tasks set the parent’s state to “Failed”, and successful completion of the final task in the chain will set the parent’s state to “Succeeded” as well.

Here’s a typical example of creating a chain with status tracking:

from celery import chain
from my_app.tasks import user_task_1, user_task_2, user_task_3

# Argument preparation omitted

name = 'Prepare report for {}'.format(arg1)
chain(user_task_1.si(arg1, user_task_name=name), user_task_2.si(arg1), user_task_3.si(arg2))

Groups#

A group is a set of Celery tasks which can be run in parallel. (Whether or not they actually run in parallel depends on the number of tasks in the group, the number of worker processes in use, and how busy those workers are processing other tasks.) django-user-tasks handles groups much as it handles chains, creating a parent UserTaskStatus record to represent the group as a whole and setting it as the parent of each contained subclass of UserTaskMixin. Most of what was said above for the status records involved in a chain also applies for groups, with the following exceptions:

  • task_class is “celery.group” for the parent status record.

  • Calling UserTaskStatus.set_state() on a child task does not affect the status of the group (because it contains multiple tasks which could be in different states at the same time).

  • state is set to “Succeeded” when all of the child tasks have succeeded.

The code for creating a group is almost identical to that for creating a chain:

from celery import group
from my_app.tasks import user_task_1, user_task_2, user_task_3

# Argument preparation omitted

name = 'Prepare report for {}'.format(arg1)
group(user_task_1.si(arg1, user_task_name=name), user_task_2.si(arg1), user_task_3.si(arg2))

Chords#

A chord is essentially a common special case of nesting a group in a chain. It consists of a “header” of one or more tasks which can be executed concurrently, followed by a “body” task which is to be started only after all of the tasks in the header have completed. django-user-tasks essentially treats it like a chain, with the following differences:

  • task_class is “celery.chord” for the parent status record.

  • Another status record with “celery.group” as the task_class is created as a child of the chord status and parent of the header tasks.

  • Calling UserTaskStatus.set_state() on the body task propagates to the parent status, but calling it for a header task does not (for the same reason given above for groups).

  • state is set to “Succeeded” when the body task succeeds.

An example of creating a chord with status tracking:

from celery import chord
from my_app.tasks import user_task_1, user_task_2, user_task_3

# Argument preparation omitted

name = 'Prepare report for {}'.format(arg1)
chord([user_task_1.si(arg1, user_task_name=name), user_task_2.si(arg1)])(user_task_3.si(arg2))

Nested Groupings#

Celery supports nesting chains, groups, and chords; you can have a group of tasks of which one or more are actually chains of other tasks, etc. While such nested constructs can probably be correctly supported in django-user-tasks, they haven’t been explicitly tested yet as they seem to be pretty rarely used in practice.