Routing tasks

Justin tcpandip at gmail.com
Wed Jul 17 16:59:03 EDT 2013


Opps, my bad, wrong list. :)


On Wed, Jul 17, 2013 at 4:54 PM, Justin <tcpandip at gmail.com> wrote:

> Is it possible to route a task to a specific worker? I looked at using
> priorities but I don't think that is an option/implemented.
>
> - I have 3 celery servers for a total of 55 workers/threads: hostA/13,
> hostB/21, and hostC/21 (concurrency is after the /)
> - I'm using RabbitMQ
> - I'm ETL'ing ~2TB of MySQL data to Riak.
> - Two workers/threads can't modify the same data at once, so I have a
> locking mechanism.
> - The transformation/merging of data into Riak can take several minutes.
> If 2 or more tasks are working on the same "key", the other has to wait
> until the merging is complete. This is very common because I create the
> tasks sequentially based off key. This often leads to many or all workers
> waiting/idle (for many minutes ... it some cases, several hours).
>
> My logic is that if I could route a "key" to a specific worker/thread, I
> would eliminate the locking.
>
> Thanks for any help.
>
> Kind regards,
> -Justin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130717/18e087a5/attachment.html>


More information about the riak-users mailing list