Riak CS

Kelly McLaughlin kelly at basho.com
Tue Jul 15 17:30:58 EDT 2014


Mohan,

I might be a bit confused on what your intent is, but it sounds like your task is to download a large group of files from S3 for processing and you are considering Riak CS for that processing work. If that is the case I am not sure Riak CS is the right fit for that job. Riak itself has a map reduce system built into it, but that is not exposed by Riak CS. Currently, Riak CS is strictly for storage and not data processing. If that is your need you might be better off looking at tools built for data processing like Spark or Hadoop. 

Kelly

On July 11, 2014 at 10:13:35 PM, Mohan Radhakrishnan (radhakrishnan.mohan at gmail.com) wrote:

I thought the general idea floating in my ignorant mind can help. We have other storage systems apart from S3 like FTPS, HTTPS etc. I thought the task of crawling remote storage systems and processing files naturally lent itself to distributed MR jobs and a DFS.

That is when I came across Riak CS.

Thanks,
Mohan


On Sat, Jul 12, 2014 at 9:32 AM, Mohan Radhakrishnan <radhakrishnan.mohan at gmail.com> wrote:
Hi,
             I came across the storage system discussion thread. We have a requirement to download thousands of files from S3 for processing. Ours is not a cloud storage system but a cloud access system.

Are there  qualities of Riak CS that can help us ? We want to download part of a huge file and checkpoint if the connection breaks. File downloads should be fault-tolerant when nodes go down.

Please bear with me if my question is basic. I don't work with distributed cloud storage systems.

Thanks,
Mohan

_______________________________________________  
riak-users mailing list  
riak-users at lists.basho.com  
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140715/f9ffd3aa/attachment.html>


More information about the riak-users mailing list