RIAK-CS crashing with many HEAD requests

Kelly McLaughlin kelly at basho.com
Wed Apr 10 19:42:49 EDT 2013


Stefan,

Thanks for the report. We will investigate this and let you know what we
find.

Kelly

On Wed, Apr 10, 2013 at 5:05 AM, Silasi Stefan <silasistefan at gmail.com>wrote:

> Hello,
>
> I'm testing RIAK-CS as an S3 alternative and I get some strange crashing
> when reading data from it. The following is reproducible with each run of
> the python script.
>
> conn = boto.connect_s3(
>     aws_access_key_id = access_key,
>     aws_secret_access_key = secret_key,
>     host = riak_host,
>     proxy = riak_host,
>     proxy_port = 8080,
>     calling_format = boto.s3.connection.OrdinaryCallingFormat(),
>     debug = 2,
>     is_secure = False,
> )
>
> bucket = conn.create_bucket('bucket_name')
>
> for row in rows:
>     key=bucket.new_key(valid_key_name)
>     if key.exists():
>         key.open_read()
>
>
> If I monitor the riak-cs memory usage, it is growing and growing until
> it's using all RAM+SWAP, then it crashes.
> In the console log, the following line is appended before RIAK-CS crashes:
>
> 2013-04-10 08:28:40.802 [info] <0.41.0> alarm_handler:
> {set,{system_memory_high_watermark,[]}}
>
> Nothing related in the error.log.
> In access.log I see a lot of HEAD requests for the objects stored.
>
> Current architecture looks like this:
> - 3 virtual machines with 4 cores, 4gb ram, 2gb swap and 1tb disk each
> - RIAK installed on all of them, configured as cluster
> - stanchion installed on all of them
> - RIAK-CS installed on all of them, configured to "speak" only with local
> RIAK
> - about 10GB of data storred in each RIAK with N=3.
>
> The strange thing is that if I move the connect-to part in the FOR loop,
> RIAK-CS memory usage doesn't go over 300MB. Any ideas what could be wrong?
>
> Thank you,
> Stefan.
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130410/dd45ec36/attachment.html>


More information about the riak-users mailing list