keys=stream

Andrew Harvey Andrew.Harvey at lexer.com.au
Wed May 12 22:17:15 EDT 2010


I've just run into this one as well.

Not sure if it's a curb thing or what, but it's making me a sad panda. We have 500k+ keys in our bucket, and I'm being returned chunks of invalid json so nothing is being yielded to my block. Needless to say, the task I'm doing here I'd prefer to do in a MapReduce, but unfortunately I need to get everything out into ruby-land for the moment.

Andrew


On 04/05/2010, at 11:46 PM, Sean Cribbs wrote:

This looks like a bug. I'll add an issue to the github tracker.

Sean Cribbs <sean at basho.com<mailto:sean at basho.com>>
Developer Advocate
Basho Technologies, Inc.
http://basho.com/

On May 4, 2010, at 9:29 AM, Adam Hunter wrote:

Just curb, I couldn't get net/http to work when doing chunked (something about read_body being called twice).


On May 4, 2010, at 9:28 AM, Sean Cribbs wrote:

Are you using curb or net/http?

Sean Cribbs <sean at basho.com<mailto:sean at basho.com>>
Developer Advocate
Basho Technologies, Inc.
http://basho.com/

On May 4, 2010, at 9:26 AM, Adam Hunter wrote:

The problem I was running into was some of the chunks were only part of the json object.  Decoded it would be something like this:

chunk 1: {'keys':[1,2,3,4
chunk 2: 5,6,7,8,9]}

I was running this on a bucket with about 11k keys.  Using ripple I could only get about 150 keys back this way because all the other chunks were not valid json.  I hacked around this for now by checking to see if the chunk ends in ]}, otherwise I accumulate chunks until I have a full json object.

Thanks,

Adam


On May 4, 2010, at 9:22 AM, Sean Cribbs wrote:

This behavior is unchanged since its implementation in 0.8 or 0.9.  To better demonstrate what that statement means, here's the joined output of a streamed keys response:

{"props":{"name":"foo","n_val":3,"allow_mult":false,"precommit":[],"postcommit":[],"chash_keyfun":{"mod":"riak_core_util","fun":"chash_std_keyfun"},"linkfun":{"mod":"riak_kv_wm_link_walker","fun":"mapreduce_linkfun"},"old_vclock":86400,"young_vclock":20,"big_vclock":50,"small_vclock":10}}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}

If your client library supports per-chunk processing, each of those {"keys":[]} objects will be an encoded chunk.  The Ruby client and most of the other clients support this.

Sean Cribbs <sean at basho.com<mailto:sean at basho.com>>
Developer Advocate
Basho Technologies, Inc.
http://basho.com/

On May 3, 2010, at 9:28 PM, Adam Hunter wrote:

Hi All,

I apologize if this has been covered already.

In the wiki it says "If keys=stream, the response will be transferred using chunked-encoding, where each chunk is a JSON object" (on http://wiki.basho.com/display/RIAK/REST+API)

This doesn't seem to be the case, sometimes a json object is broken into several chunks.  Is this behavior correct?

Thanks,

Adam
_______________________________________________
riak-users mailing list
riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com








Click here<https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg==> to report this email as spam.

_______________________________________________
riak-users mailing list
riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


Andrew Harvey / Developer
lexer

m/
t/      +61 2 9019 6379
w/      http://lexer.com.au

Help put an end to whaling. Visit www.givewhalesavoice.com.au<http://www.givewhalesavoice.com.au/>

________________________________



Please consider the environment before printing this email
This email transmission is confidential and intended solely for the person or organisation to whom it is addressed. If you are not the intended recipient, you must not copy, distribute or disseminate the information, or take any action in relation to it and please delete this e-mail. Any views expressed in this message are those of the individual sender, except where the send specifically states them to be the views of any organisation or employer. If you have received this message in error, do not open any attachment but please notify the sender (above). This message has been checked for all known viruses powered by McAfee.

For further information visit http://www.mcafee.com/us/threat_center/default.asp
Please rely on your own virus check as no responsibility is taken by the sender for any damage rising out of any virus infection this communication may contain.


This message has been scanned for malware by Websense. www.websense.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20100513/eefbe8ec/attachment-0001.html>


More information about the riak-users mailing list