RIAK-CS crashing with many HEAD requests

Silasi Stefan silasistefan at gmail.com
Wed Apr 10 07:05:53 EDT 2013


Hello,

I'm testing RIAK-CS as an S3 alternative and I get some strange crashing
when reading data from it. The following is reproducible with each run of
the python script.

conn = boto.connect_s3(
    aws_access_key_id = access_key,
    aws_secret_access_key = secret_key,
    host = riak_host,
    proxy = riak_host,
    proxy_port = 8080,
    calling_format = boto.s3.connection.OrdinaryCallingFormat(),
    debug = 2,
    is_secure = False,
)

bucket = conn.create_bucket('bucket_name')

for row in rows:
    key=bucket.new_key(valid_key_name)
    if key.exists():
        key.open_read()


If I monitor the riak-cs memory usage, it is growing and growing until it's
using all RAM+SWAP, then it crashes.
In the console log, the following line is appended before RIAK-CS crashes:

2013-04-10 08:28:40.802 [info] <0.41.0> alarm_handler:
{set,{system_memory_high_watermark,[]}}

Nothing related in the error.log.
In access.log I see a lot of HEAD requests for the objects stored.

Current architecture looks like this:
- 3 virtual machines with 4 cores, 4gb ram, 2gb swap and 1tb disk each
- RIAK installed on all of them, configured as cluster
- stanchion installed on all of them
- RIAK-CS installed on all of them, configured to "speak" only with local
RIAK
- about 10GB of data storred in each RIAK with N=3.

The strange thing is that if I move the connect-to part in the FOR loop,
RIAK-CS memory usage doesn't go over 300MB. Any ideas what could be wrong?

Thank you,
Stefan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130410/775c6c53/attachment.html>


More information about the riak-users mailing list