Python background job patterns including task queues, workers, and event-driven architecture. Use when implementing async task processing, job queues, long-running operations, or decoupling work from request/response cycles.

Python Background Jobs & Task Queues Decouple long running or unreliable work from request/response cycles. Return immediately to the user while background workers handle the heavy lifting asynchronously. When to Use This Skill Processing tasks that take longer than a few seconds Sending emails, notifications, or webhooks Generating reports or exporting data Processing uploads or media transformations Integrating with unreliable external services Building event driven architectures Core Concepts 1. Task Queue Pattern API accepts request, enqueues a job, returns immediately with a job ID. Workers process jobs asynchronously. 2. Idempotency Tasks may be retried on failure. Design for safe re execution. 3. Job State Machine Jobs transition through states: pending → running → succeeded/failed. 4. At Least Once Delivery Most queues guarantee at least once delivery. Your code must handle duplicates. Quick Start This skill uses Celery for examples, a widely adopted task queue. Alternatives like RQ, Dramatiq, and cloud native solutions (AWS SQS, GCP Tasks) are equally valid choices. Fundamental Patterns Pattern 1: Return Job ID Immediately For operations exceeding a few seconds, return a job ID and process asynchronously. Pattern 2: Celery Task Configuration Configure Celery tasks with proper retry and timeout settings. Pattern 3: Make Tasks Idempotent Workers may retry on crash or timeout. Design for safe re execution. Idempotency Strategies: 1. Check before write : Verify state before action 2. Idempotency keys : Use unique tokens with external services 3. Upsert patterns : INSERT ... ON CONFLICT UPDATE 4. Deduplication window : Track processed IDs for N hours Pattern 4: Job State Management Persist job state transitions for visibility and debugging. Detailed worked examples and patterns Detailed sections (starting with Advanced Patterns ) live in references/details.md . Read that file when the navigation summary above is insufficient. Best Practices Summary 1. Return immediately Don't block requests for long operations 2. Persist job state Enable status polling and debugging 3. Make tasks idempotent Safe to retry on any failure 4. Use idempotency keys For external service calls 5. Set timeouts Both soft and hard limits 6. Implement DLQ Capture permanently failed tasks 7. Log transitions Track job state changes 8. Retry appropriately Exponential backoff for transient errors 9. Don't retry permanent failures Validation errors, invalid credentials 10. Monitor queue depth Alert on backlog growth

python-background-jobs