keys=stream

Sean Cribbs sean at basho.com
Wed May 12 23:17:33 EDT 2010


If the problem is with net/http then we need to fix that backend, not apply hacks to the generic portions.  You can verify that Riak will return valid JSON objects in each chunk by adding --raw to the curl command.  Here's what I get after loading the Fast Track stock data: http://gist.github.com/399450 

Sean Cribbs <sean at basho.com>
Developer Advocate
Basho Technologies, Inc.
http://basho.com/

On May 12, 2010, at 10:55 PM, Adam Hunter wrote:

> This is a confirmed bug and will be fixed soon.  You can check out http://github.com/adamhunter/ripple/blob/associations/lib/riak/bucket.rb : 65 for a temporary hack.
> 
> Thanks,
> 
> Adam
> 
> On May 12, 2010, at 10:17 PM, Andrew Harvey wrote:
> 
>> I've just run into this one as well.
>> 
>> 
>> Not sure if it's a curb thing or what, but it's making me a sad panda. We have 500k+ keys in our bucket, and I'm being returned chunks of invalid json so nothing is being yielded to my block. Needless to say, the task I'm doing here I'd prefer to do in a MapReduce, but unfortunately I need to get everything out into ruby-land for the moment.
>> 
>> Andrew
>> 
>> 
>> On 04/05/2010, at 11:46 PM, Sean Cribbs wrote:
>> 
>>> This looks like a bug. I'll add an issue to the github tracker.
>>> 
>>> Sean Cribbs <sean at basho.com>
>>> Developer Advocate
>>> Basho Technologies, Inc.
>>> http://basho.com/
>>> 
>>> On May 4, 2010, at 9:29 AM, Adam Hunter wrote:
>>> 
>>>> Just curb, I couldn't get net/http to work when doing chunked (something about read_body being called twice).
>>>> 
>>>> 
>>>> On May 4, 2010, at 9:28 AM, Sean Cribbs wrote:
>>>> 
>>>>> Are you using curb or net/http?
>>>>> 
>>>>> Sean Cribbs <sean at basho.com>
>>>>> Developer Advocate
>>>>> Basho Technologies, Inc.
>>>>> http://basho.com/
>>>>> 
>>>>> On May 4, 2010, at 9:26 AM, Adam Hunter wrote:
>>>>> 
>>>>>> The problem I was running into was some of the chunks were only part of the json object.  Decoded it would be something like this:
>>>>>> 
>>>>>> chunk 1: {'keys':[1,2,3,4
>>>>>> chunk 2: 5,6,7,8,9]}
>>>>>> 
>>>>>> I was running this on a bucket with about 11k keys.  Using ripple I could only get about 150 keys back this way because all the other chunks were not valid json.  I hacked around this for now by checking to see if the chunk ends in ]}, otherwise I accumulate chunks until I have a full json object.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Adam
>>>>>> 
>>>>>> 
>>>>>> On May 4, 2010, at 9:22 AM, Sean Cribbs wrote:
>>>>>> 
>>>>>>> This behavior is unchanged since its implementation in 0.8 or 0.9.  To better demonstrate what that statement means, here's the joined output of a streamed keys response:
>>>>>>> 
>>>>>>> {"props":{"name":"foo","n_val":3,"allow_mult":false,"precommit":[],"postcommit":[],"chash_keyfun":{"mod":"riak_core_util","fun":"chash_std_keyfun"},"linkfun":{"mod":"riak_kv_wm_link_walker","fun":"mapreduce_linkfun"},"old_vclock":86400,"young_vclock":20,"big_vclock":50,"small_vclock":10}}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}{"keys":[]}
>>>>>>> 
>>>>>>> If your client library supports per-chunk processing, each of those {"keys":[]} objects will be an encoded chunk.  The Ruby client and most of the other clients support this.
>>>>>>> 
>>>>>>> Sean Cribbs <sean at basho.com>
>>>>>>> Developer Advocate
>>>>>>> Basho Technologies, Inc.
>>>>>>> http://basho.com/
>>>>>>> 
>>>>>>> On May 3, 2010, at 9:28 PM, Adam Hunter wrote:
>>>>>>> 
>>>>>>>> Hi All,
>>>>>>>> 
>>>>>>>> I apologize if this has been covered already.
>>>>>>>> 
>>>>>>>> In the wiki it says "If keys=stream, the response will be transferred using chunked-encoding, where each chunk is a JSON object" (on http://wiki.basho.com/display/RIAK/REST+API)
>>>>>>>> 
>>>>>>>> This doesn't seem to be the case, sometimes a json object is broken into several chunks.  Is this behavior correct?
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Adam
>>>>>>>> _______________________________________________
>>>>>>>> riak-users mailing list
>>>>>>>> riak-users at lists.basho.com
>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> Click here to report this email as spam.
>>> 
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
>> 
>> Andrew Harvey / Developer 
>> lexer
>> 
>> m/ 	
>> t/ 	+61 2 9019 6379
>> w/  	http://lexer.com.au
>> 
>> Help put an end to whaling. Visit www.givewhalesavoice.com.au
>> 
>>  
>> 
>> Please consider the environment before printing this email
>> This email transmission is confidential and intended solely for the person or organisation to whom it is addressed. If you are not the intended recipient, you must not copy, distribute or disseminate the information, or take any action in relation to it and please delete this e-mail. Any views expressed in this message are those of the individual sender, except where the send specifically states them to be the views of any organisation or employer. If you have received this message in error, do not open any attachment but please notify the sender (above). This message has been checked for all known viruses powered by McAfee.
>>  
>> For further information visit http://www.mcafee.com/us/threat_center/default.asp
>> Please rely on your own virus check as no responsibility is taken by the sender for any damage rising out of any virus infection this communication may contain.
>> 
>> 
>> This message has been scanned for malware by Websense. www.websense.com
>> 
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20100512/e66cfd8f/attachment.html>


More information about the riak-users mailing list