Celery is an
open source
Open source is source code that is made freely available for possible modification and redistribution. Products include permission to use and view the source code, design documents, or content of the product. The open source model is a decentrali ...
asynchronous
task queue or job queue which is based on distributed message passing. While it supports scheduling, its focus is on operations in real time.
Overview
The execution units, called ''tasks'', are executed concurrently on one or more worker nodes using
multiprocessing
Multiprocessing (MP) is the use of two or more central processing units (CPUs) within a single computer system. The term also refers to the ability of a system to support more than one processor or the ability to allocate tasks between them. The ...
, or . Tasks can execute asynchronously (in the background) or synchronously (wait until ready). Celery is used in production systems, for services such as
Instagram
Instagram is an American photo sharing, photo and Short-form content, short-form video sharing social networking service owned by Meta Platforms. It allows users to upload media that can be edited with Social media camera filter, filters, be ...
, to process millions of tasks every day.
Technology
Celery is written in
Python
Python may refer to:
Snakes
* Pythonidae, a family of nonvenomous snakes found in Africa, Asia, and Australia
** ''Python'' (genus), a genus of Pythonidae found in Africa and Asia
* Python (mythology), a mythical serpent
Computing
* Python (prog ...
, but the protocol can be implemented in any language. It can also operate with other languages using
webhook
In web development, a webhook is a method of augmenting or altering the behavior of a web page or web application with custom callbacks. These callbacks may be maintained, modified, and managed by third-party users who need not be affiliated with ...
s. There is also a
Ruby
Ruby is a pinkish-red-to-blood-red-colored gemstone, a variety of the mineral corundum ( aluminium oxide). Ruby is one of the most popular traditional jewelry gems and is very durable. Other varieties of gem-quality corundum are called sapph ...
-Client called RCelery, a
PHP
PHP is a general-purpose scripting language geared towards web development. It was originally created by Danish-Canadian programmer Rasmus Lerdorf in 1993 and released in 1995. The PHP reference implementation is now produced by the PHP Group. ...
client, a
Go client, a
Rust
Rust is an iron oxide, a usually reddish-brown oxide formed by the reaction of iron and oxygen in the catalytic presence of water or air moisture. Rust consists of hydrous iron(III) oxides (Fe2O3·nH2O) and iron(III) oxide-hydroxide (FeO(OH) ...
client, and a
Node.js client.
Celery requires a
message broker
A message broker (also known as an integration broker or interface engine) is an intermediary computer program module that translates a message from the formal messaging protocol of the sender to the formal messaging protocol of the receiver. Mes ...
to run. As of October 2024,
Redis
Redis (; Remote Dictionary Server) is an in-memory key–value database, used as a distributed cache and message broker, with optional durability. Because it holds all data in memory and because of its design, Redis offers low- latency reads ...
and
RabbitMQ
RabbitMQ is an open-source message-broker software (sometimes called message-oriented middleware) that originally implemented the Advanced Message Queuing Protocol (AMQP) and has since been extended with a plug-in architecture to support Str ...
are supported and actively maintained and monitored.
Amazon SQS
Amazon Simple Queue Service (Amazon SQS) is a distributed message queuing service introduced by Amazon.com as a beta in late 2004, and generally available in mid 2006.
API
Amazon provides SDKs in several programming languages, including:
* C ...
is supported and maintained but does not support worker inspection and management at runtime, while Zookeeper and Kafka are currently in experimental development.
See also
*
Advanced Message Queuing Protocol
The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware. The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subs ...
References
External links
*
*
Free system software
Free software programmed in Python
Python (programming language) software
Concurrent programming libraries
Inter-process communication
Message-oriented middleware
Software using the BSD license
{{free-software-stub