Importing data to Riak

Russell Brown russelldb at basho.com
Mon Nov 14 08:14:21 EST 2011


On 14 Nov 2011, at 11:47, Nitish Sharma wrote:

> Hi,
> This is more sort of a discussion than a question. I am just trying to see the trend in how users import their data to Riak. 
> For the data I am using, I was able to achieve almost 150 records/second with PHP library, and 400 records/second with node.js (fairly new with node; was hitting memory wall when trying to import 1 million records).
> What are some hacks/tricks/tweaks to import large amount of data to Riak?

New keys, new data, straight in for the first time, no fetch before store? I've had reasonable results creating a *number* of threads and using the Java Raw PB client to write. 

For example, maybe have a 1 or a couple of threads that reads data (from Oracle, a file, what-have-you) and puts it on a queue, and have a bunch of threads that pull data off the queue, create a riak object and store it. From my laptop I've got up to 2500 writes a second like this, and it was just ad hoc, throw away code with 4 threads against a small 3 node cluster (running on desktops.)

I imagine others on the list have more direct, real world examples?

Cheers

Russell

> 
> Cheers
> Nitish
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





More information about the riak-users mailing list