Routing tasks

Justin tcpandip at gmail.com
Wed Jul 17 16:54:50 EDT 2013


Is it possible to route a task to a specific worker? I looked at using
priorities but I don't think that is an option/implemented.

- I have 3 celery servers for a total of 55 workers/threads: hostA/13,
hostB/21, and hostC/21 (concurrency is after the /)
- I'm using RabbitMQ
- I'm ETL'ing ~2TB of MySQL data to Riak.
- Two workers/threads can't modify the same data at once, so I have a
locking mechanism.
- The transformation/merging of data into Riak can take several minutes. If
2 or more tasks are working on the same "key", the other has to wait until
the merging is complete. This is very common because I create the tasks
sequentially based off key. This often leads to many or all workers
waiting/idle (for many minutes ... it some cases, several hours).

My logic is that if I could route a "key" to a specific worker/thread, I
would eliminate the locking.

Thanks for any help.

Kind regards,
-Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130717/fc7bcefa/attachment.html>


More information about the riak-users mailing list