Importing data to Riak
russelldb at basho.com
Mon Nov 14 08:14:21 EST 2011
On 14 Nov 2011, at 11:47, Nitish Sharma wrote:
> This is more sort of a discussion than a question. I am just trying to see the trend in how users import their data to Riak.
> For the data I am using, I was able to achieve almost 150 records/second with PHP library, and 400 records/second with node.js (fairly new with node; was hitting memory wall when trying to import 1 million records).
> What are some hacks/tricks/tweaks to import large amount of data to Riak?
New keys, new data, straight in for the first time, no fetch before store? I've had reasonable results creating a *number* of threads and using the Java Raw PB client to write.
For example, maybe have a 1 or a couple of threads that reads data (from Oracle, a file, what-have-you) and puts it on a queue, and have a bunch of threads that pull data off the queue, create a riak object and store it. From my laptop I've got up to 2500 writes a second like this, and it was just ad hoc, throw away code with 4 threads against a small 3 node cluster (running on desktops.)
I imagine others on the list have more direct, real world examples?
> riak-users mailing list
> riak-users at lists.basho.com
More information about the riak-users