2i timeouts in 1.4

Russell Brown russell.brown at mac.com
Sat Jul 27 11:28:57 EDT 2013


Hi Sean,

This one really is a bug with 2i pagination on Equals term queries[1].

I've fixed it already[2], I'm very sorry that this one got out into the wild. It is something that worked then stopped working at some point in a merge. It was a pretty bad oversight that we didn't have a test for this condition to catch the regression.

I've added a test[3] to the suite for the issue and verified the patch. I guess there will be a 1.4.1 soon.

Apologies, again.

Russell

[1] https://github.com/basho/riak_kv/issues/611
[2] https://github.com/basho/riak_kv/pull/612
[3] https://github.com/basho/riak_test/pull/340

On 27 Jul 2013, at 01:11, Sean McKibben <graphex at graphex.com> wrote:

> So when I try to use pagination, it doesn't seem to be picking up my continuation. I'm having trouble parsing the json I get back using stream=true (and there is still a timeout) so I went to just using pagination. Perhaps I'm doing it wrong, (likely, it has been a long day) but riak seems to be ignoring my continuation:
> 
> (pardon the sanitization)
> curl 'http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?max_results=5'
> {"keys":["1","2","3","4","5"],"continuation":"g20AAABAMDAwMDE1ZWVjMmNiZjY3Y2Y4YmU3ZTVkMWNiZTVjM2ZkYjg2YWU0MGIwNzNjMTE3NDYyZjEzMTNlMDQ5YmI2ZQ=="}
> 
> curl 'http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?max_results=5&continuation=g20AAABAMDAwMDE1ZWVjMmNiZjY3Y2Y4YmU3ZTVkMWNiZTVjM2ZkYjg2YWU0MGIwNzNjMTE3NDYyZjEzMTNlMDQ5YmI2ZQ=='
> {"keys":["1","2","3","4","5"],"continuation":"g20AAABAMDAwMDE1ZWVjMmNiZjY3Y2Y4YmU3ZTVkMWNiZTVjM2ZkYjg2YWU0MGIwNzNjMTE3NDYyZjEzMTNlMDQ5YmI2ZQ=="}
> 
> The same keys and continuation value are returned regardless of whether my request contains a continuation value. I've tried swapping the order of max_results and continuation without any luck. I also made sure that my continuation value was url encoded. Hopefully I'm not missing something obvious here. Well, come to think of it, hopefully I am missing something obvious!
> 
> Sean
> 
> On Jul 26, 2013, at 6:43 PM, Russell Brown <russell.brown at mac.com> wrote:
> 
>> For a work around you could use streaming and pagination.
>> 
>> Request smaller pages of data (i.e. sub 60 seconds worth) and use streaming to get the results to your client sooner.
>> 
>> In HTTP this would look like
>> 
>> http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?max_results=10000&stream=true
>> 
>> your results will include a continuation like
>> 
>>   {"continuation":"g2gCYgAAFXttAAAABDU0OTk="}
>> 
>> and you can use that to get the next N results. Breaking your query up that way should duck the timeout.
>> 
>> Furthermore, adding &stream=true will mean the first results is received very rapidly.
>> 
>> I don't think the Ruby client is up to date for the new 2i features, but you could monkeypatch as before.
>> 
>> Cheers
>> 
>> Russell
>> 
>> On 26 Jul 2013, at 19:00, Sean McKibben <graphex at graphex.com> wrote:
>> 
>>> Thank you for looking in to this. This is a major problem for our production cluster, and we're in a bit of a bind right now trying to figure out a workaround in the interim. It sounds like maybe a mapreduce might handle the timeout properly, so hopefully we can do that in the meantime.
>>> If there is any way we can have a hotfix ASAP though, that would be preferable. Certainly would not be a problem for us to edit a value in the config file (and given the lack of support in the ruby client for the timeout setting, the ability to edit the global default would be preferred).
>>> In the ruby client i had to monkeypatch it like this to even submit the timeout value, which is not ideal:
>>> 
>>> module Riak
>>> class Client
>>>  class HTTPBackend
>>>    def get_index(bucket, index, query)
>>>      bucket = bucket.name if Bucket === bucket
>>>      path = case query
>>>             when Range
>>>               raise ArgumentError, t('invalid_index_query', :value => query.inspect) unless String === query.begin || Integer === query.end
>>>               index_range_path(bucket, index, query.begin, query.end)
>>>             when String, Integer
>>>               index_eq_path(bucket, index, query, 'timeout' => '260000')
>>>             else
>>>               raise ArgumentError, t('invalid_index_query', :value => query.inspect)
>>>             end
>>>      response = get(200, path)
>>>      JSON.parse(response[:body])['keys']
>>>    end
>>>  end
>>> end
>>> end
>>> 
>>> Thanks for the update,
>>> Sean
>>> 
>>> 
>>> 
>>> On Jul 26, 2013, at 4:49 PM, Russell Brown <russell.brown at mac.com> wrote:
>>> 
>>>> Hi Sean,
>>>> I'm very sorry to say that you've found a featurebug.
>>>> 
>>>> There was a fix put in here https://github.com/basho/riak_core/pull/332
>>>> 
>>>> But that means that the default timeout of 60 seconds is now honoured. In the past it was not.
>>>> 
>>>> As far as I can see the 2i endpoint never accepted a timeout argument, and it still does not.
>>>> 
>>>> The fix would be to add the timeout to the 2i API endpoints, and I'll do that straight away.
>>>> 
>>>> In the meantime, I wonder if streaming the results would help, or if you'd still hit the overall timeout?
>>>> 
>>>> Very sorry that you've run into this. Let me know if streaming helps, I've raised an issue here[1] if you want to track this bug
>>>> 
>>>> Cheers
>>>> 
>>>> Russell
>>>> 
>>>> [1] https://github.com/basho/riak_kv/issues/610
>>>> 
>>>> 
>>>> On 26 Jul 2013, at 17:59, Sean McKibben <graphex at graphex.com> wrote:
>>>> 
>>>>> I should have mentioned that I also tried:
>>>>> curl -H "X-Riak-Timeout: 260000" "http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?timeout=260000" -i
>>>>> but still receive the 500 error below exactly at the 60 second mark. Is this a bug?
>>>>> 
>>>>> Secondary to getting this working at all, is this documented anywhere? and any way to set this timeout using the ruby riak client?
>>>>> 
>>>>> Stream may well work, but I'm going to have to make a number of changes on the client side to deal with the results.
>>>>> 
>>>>> Sean
>>>>> 
>>>>> On Jul 26, 2013, at 3:53 PM, Brian Roach <roach at basho.com> wrote:
>>>>> 
>>>>>> Sean -
>>>>>> 
>>>>>> The timeout isn't via a header, it's a query param -> &timeout=xxxx
>>>>>> 
>>>>>> You can also use stream=true to stream the results.
>>>>>> 
>>>>>> - Roach
>>>>>> 
>>>>>> Sent from my iPhone
>>>>>> 
>>>>>> On Jul 26, 2013, at 3:43 PM, Sean McKibben <graphex at graphex.com> wrote:
>>>>>> 
>>>>>>> We just upgraded to 1.4 and are having a big problem with some of our larger 2i queries. We have a few key queries that takes longer than 60 seconds (usually about 110 seconds) to execute, but after going to 1.4 we can't seem to get around a 60 second timeout.
>>>>>>> 
>>>>>>> I've tried:
>>>>>>> curl -H "X-Riak-Timeout: 260000" "http://127.0.0.1:8098/buckets/mybucket/index/test_bin/myval?x-riak-timeout=260000" -i
>>>>>>> 
>>>>>>> But I always get
>>>>>>> HTTP/1.1 500 Internal Server Error
>>>>>>> Vary: Accept-Encoding
>>>>>>> Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
>>>>>>> Date: Fri, 26 Jul 2013 21:41:28 GMT
>>>>>>> Content-Type: text/html
>>>>>>> Content-Length: 265
>>>>>>> Connection: close
>>>>>>> 
>>>>>>> <html><head><title>500 Internal Server Error</title></head><body><h1>Internal Server Error</h1>The server encountered an error while processing this request:<br><pre>{error,{error,timeout}}</pre><P><HR><ADDRESS>mochiweb+webmachine web server</ADDRESS></body></html>
>>>>>>> 
>>>>>>> Right at the 60 second mark. What can I set to give my secondary index queries more time??
>>>>>>> 
>>>>>>> This is causing major problems for us :(
>>>>>>> 
>>>>>>> Sean
>>>>>>> _______________________________________________
>>>>>>> riak-users mailing list
>>>>>>> riak-users at lists.basho.com
>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>> 
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> riak-users at lists.basho.com
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>> 
>>> 
>> 
>> 
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 





More information about the riak-users mailing list