Massive json schema update

Damien Krotkine damien.krotkine at booking.com
Mon Jun 13 05:25:52 EDT 2016


My advice : use the fetch-update loop that you already know. You won't 
get better performance or reliability by using a MapReduce.

I understand that your values are indexed in Search. So either use a 
search query and update them all, or you could also do a list_keys in 
stream mode, and updated the values.


Guillaume Boddaert wrote:
>
> Well,
>
> We are talking about ~30 million items, in a single bucket. Those are
> standard tweets, I need to alter all of them because of an invalid
> structure.
> My cluster is not in production, but I am willing to start validating
> a procedure that will be used at some point on a production cluster,
> that's a pre-release rehearsal.
>
> I can write my own script that read/update every entries in big scan
> of the bucket, but It may suffer from network lag, or other external
> stuffs. I'd really like to learn how to alter a set of document in the
> erlang console for instance, in order to validate that I can patch
> data easily in something that would looks more like an SQL command
> (Yes, that's not how it works, understand a map-reduce script in a
> mongo shell for instance) in the Riak core than a programmatic
> fetch/update loop that I already know.
>
> Guillaume
>
> On 08/06/2016 08:49, Damien Krotkine wrote:
>>
>> Hi Guillaume,
>>
>> If I understand correctly you need to change all the values of your
>> JSON data.
>>
>> How many keys are we talking about, how big are the data, and in how
>> many buckets are the keys?
>>
>> Also, is your cluster in production yet?
>>
>>
>>
>>>
>>> Le 7 juin 2016 à 18:43, Guillaume Boddaert
>>> <guillaume at lighthouse-analytics.co> a écrit :
>>>
>>> Hi,
>>>
>>> I'd like to patch my current riak collection to rename a field
>>> inside a JSON schema, how can I achieve that from command line on
>>> the riak server itself ? Is there some kind of map/reduce mechanism
>>> that allow any json record to be updated then saved to the riak
>>> cluster ?
>>>
>>> Guillaume
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20160613/f0759fc0/attachment-0002.html>


More information about the riak-users mailing list