Download as PDF, PPTX
![A typical job/task
def send_an_email(email_type, user):
html = template(email_type, user)
status = email.send(html, user["email"])
metrics.send("email_%s" % status, 1)
return status
KERNEL PANIC](https://image.slidesharecdn.com/pyparis2017-taskqueues-170612163709/75/Developer-friendly-taskqueues-What-you-should-ask-yourself-before-choosing-one-6-2048.jpg)
![A typical job/task
def send_an_email(email_type, user):
html = template(email_type, user)
status = email.send(html, user["email"])
metrics.send("email_%s" % status, 1)
return status
KERNEL PANIC](https://crownmelresort.com/image.slidesharecdn.com/pyparis2017-taskqueues-170612163709/75/Developer-friendly-taskqueues-What-you-should-ask-yourself-before-choosing-one-6-2048.jpg)
Uploaded bySylvain Zimmer
Developer-friendly taskqueues: What you should ask yourself before choosing one
This document summarizes key considerations for choosing a task queue system. It discusses task properties like idempotency and reentrancy. It covers performance factors like latency and throughput as well as consistency models. Common task queue systems like Celery, RQ, and MRQ are evaluated based on factors like performance, complexity, community support, and future plans. The document emphasizes thinking carefully about specific needs before choosing a system and being grateful for open source software.
More Related Content
![[231] the simplicity of cluster apps with circuit](https://cdn.slidesharecdn.com/ss_thumbnails/213thesimplicityofclusterappswithcircuit-150915022150-lva1-app6892-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Developer-friendly taskqueues: What you should ask yourself before choosing one
![[fr] Introduction et Live-code Backbone.js à DevoxxFR 2013](https://cdn.slidesharecdn.com/ss_thumbnails/backbone-devoxxfr-130327123943-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Developer-friendly taskqueues: What you should ask yourself before choosing one
- 1. Sylvain Zimmer /@sylvinus PyParis 2017 DEVELOPER-FRIENDLY TASKQUEUES WHAT WE LEARNED BUILDING MRQ & WHAT YOU SHOULD ASK YOURSELF BEFORE CHOOSING ONE
- 2. /usr/bin/whoami ▸ (SpaceX nerd) ▸Founder, dotConferences ▸ CTO Pricing Assistant ▸ Co-organizer Paris.py meetup ▸ User of Python taskqueues for 10+ years ▸ Main contributor of MRQ
- 3. 4 years ago... OMG!1!!!
- 4.
- 5. Credit: Adrien DiPasquale
- 6. A typical job/task defsend_an_email(email_type, user): html = template(email_type, user) status = email.send(html, user["email"]) metrics.send("email_%s" % status, 1) return status KERNEL PANIC
- 7. Task properties Re-entrant IdempotentNullipotent< < ▸ Safe to interrupt and then retry ▸ Safe to call multiple times ▸ Result will be the same ▸ Free of side-effects def reentrant(a): value = a + random() db.insert(value) def idempotent(key, value): db.update(key, value) def nullipotent(a): return a ** 2
- 8. Other task properties& best practices ▸ Serializable args, serializable result ▸ Args validation / documentation ▸ Least args possible ▸ Canonical path vs. registration ▸ Concurrent safety ▸ Statuses
- 9. Coroutines vs. Threadsvs. Processes ▸ IO-bound tasks vs. CPU-bound tasks ▸ Threads offer few benefits for a Python worker (GIL) ▸ Coroutines/Greenlets are ideal for IO-bound tasks ▸ Processes are required for CPU-bound tasks ▸ If you have heterogenous tasks, your TQ should support both! $ mrq-worker --greenlets 25 --processes 4
- 10. Performance: latency &throughput APP BROKER WORKER RESULT STORE
- 11.
- 13. Errors ▸ Exception handlers ▸Timeouts ▸ Retry rules ▸ Sentry & friends ▸ gevent: test your tracebacks! ▸ Priorities ▸ Human process to manage failed tasks!
- 14. Task visibility ▸ Tasksby status, path, worker, ... ▸ Tracebacks & current stack ▸ Logs ▸ Timing info ▸ Cancel / Kill / Move tasks ▸ Progress
- 15. Memory leaks ▸ Workers= long-running processes ▸ gevent makes debugging harder ▸ Watch out for global variables or mutable class attributes! ▸ Python's ecosystem is surprisingly poor in this area ▸ guppy, objgraph can usually help
- 16. Misc tools ▸ Scheduler ▸Command-line runner, e.g. mrq-run tasks.myTask {"a": 1} ▸ Autoscaling ▸ Profiler
- 17.
- 18. Consistency guarantees ▸ Atleast once vs. At most once vs. Exactly once ▸ Ordering ▸ Critical operations: ▸ Queueing ▸ Marking tasks as started ▸ Timeouts & retries
- 19. Types of brokers ▸Specialized message queues (RabbitMQ, SNS, Kafka, ...) ▸ Performance, complexity, poor visibility ▸ In-memory data stores (Redis, ...) ▸ Performance, simplicity, harder to scale ▸ Regular databases (MongoDB, PostgreSQL, ...) ▸ Often enough for the job!
- 20. At the heartof the broker ▸ Atomic update from "queued" to "started" ▸ MRQ with MongoDB broker: find_one_and_update() ▸ MRQ with Redis broker: Pushback in a ZSET
- 21. ZSETs in Redis ▸Sorted sets with O(log(N)) scalability ▸ set/get by key, order by key, lookups by key or value ▸ Very interesting properties for task queues: Unicity, Ordering, Atomicity of updates, Performance ▸ MRQ's "Pushback" model: ▸ Queue with key=timestamp ▸ Unqueue by fetching key range & setting new keys in the future ▸ After completion the task adjusts or removes the key
- 23. Consistency guarantees ▸ Mustbe thought of for the whole system, not just the broker! ▸ Brokers can be misused or misconfigured ▸ The workers can drop tasks if they want to ;-) ▸ Consistency starts at queueing time!
- 24.
- 25. Think hard aboutwhat you need ▸ Will your taskqueue be the foundation of your architecture, or is it just a side project? ▸ What performance do you need? (IO vs. CPU, latency, ...) ▸ What level of visibility and control do you need on queued & running tasks? ▸ Can workers terminate abruptly? Lots of design consequences! ▸ What language interop do you need?
- 26. And then allthe usual questions... ▸ Is it supported by a lively community? ▸ License ▸ Documentation ▸ Future plans
- 27. Which one topick? ▸ Celery: High performance, large community, very complex, major upgrades painful ▸ RQ: Extremely simple to understand, low performance ▸ MRQ: Adjust task visibility vs. performance, simple to understand, 1.0 soon ▸ Lots of other valid options! Just be sure to ask yourself the right questions ;-)
- 28. BE GRATEFUL FOR THEOSS YOU USE! REMINDER
- 29.
- 30.