A semaphore-like mechanism for Celery

We're developing a distributed application in Python + Celery for our task queue.

Our application requires us to download emails from a remote ISP via IMAP (e.g.: gmail) and we're looking to have be able this task be done in parallel. For a given email account you're granted a limited to a number of simulations connections, so we need a way to atomically keep track of our active connections for all accounts being downloaded.

I've found multiple examples of atomic locks for Celery using Redis, but none that can keep track of a pool of limited resources like this, and all attempts to implement our own have resulted in difficult to debug race-conditions, causing our locks to intermittently never get released.

7
задан NFicano 3 May 2011 в 22:49
поделиться