some problems with storage statistic calculation

Stanislav Vlasov stanislav.v.v at gmail.com
Wed Aug 13 05:00:21 EDT 2014


2014-08-11 17:03 GMT+06:00 Stanislav Vlasov <stanislav.v.v at gmail.com>:

> We have 8 nodes with riak+riak-cs, about 7 Tb data in cluster.
> Some time riak process dead by OOM on several nodes when large
> (~100Gb) file was writed via s3cmd, because riak-cs eated all memory.
> After recovering we experiencing the following problems:
>
> 1) very slow storage statistic calculation (about two hour). Before
> oom it was done in 40 minutes.

After total riak-cs restart statistic calculate fine (about 10 min
from begin to end). But 15 minutes after restart statstic calculation
slowing agan and we see in console log:

2014-08-13 08:18:31.854 [warning]
<0.4043.0>@riak_cs_manifest:maybe_warn_bloated_manifests:145 Large
manifest size (54195024 bytes) for bucket=<<"test">> key=
<<"u7850.netangels.ru-20140328-full.tar.lzo">>
2014-08-13 08:18:43.620 [warning]
<0.4081.0>@riak_cs_manifest:maybe_warn_bloated_manifests:145 Many
manifest siblings (21 siblings) for bucket=<<"test">> key=
<<"u7850.netangels.ru-20140328-full.tar.lzo">>

Filesize  was ~120Gb, uploaded before OOM. File does not exists now
(deleted, uploaded with zero size and deleted again), but nothing
changed.

I guess it's a garbage collector issue, because it happen exactly
every 15 minutes ({gc_interval,900}) after restart.
What can i do with it?

-- 
Stanislav




More information about the riak-users mailing list