anti_entropy and solr taking up suspiciously large amounts of space

Rohit Sanbhadti sanbhadtirohit at vmware.com
Tue Mar 21 21:53:06 EDT 2017


Matthew,

To clarify, this happens on all nodes in our cluster (10+ nodes) although the exact size varies by 10s of GB. I’ve rolling restarted the cluster the last time this happened (last week) with no significant change in size, although the output from riak-admin aae-status and riak-admin search aae-status shows empty after the restart.  

-- 
Rohit S.

On 3/21/17, 5:25 PM, "Matthew Von-Maszewski" <matthewv at basho.com> wrote:

    Rohit,
    
    If you restart the node does the elevated anti_entropy size decline after the restart?
    
    Matthew
    
    
    > On Mar 21, 2017, at 8:00 PM, Rohit Sanbhadti <sanbhadtirohit at vmware.com> wrote:
    > 
    > Running Riak 2.2.0 on Ubuntu 16.04.1, we’ve noticed that anti_entropy is taking up way too much space on all of our nodes. We use multi_backend with mostly bitcask backends (relevant part of config pasted below). Has anyone seen this before, or does anyone have an idea what might be causing this? 
    > 
    > storage_backend = multi
    > multi_backend.bitcask_1day.storage_backend = bitcask
    > multi_backend.bitcask_1day.bitcask.data_root = /var/lib/riak/bitcask_1day
    > multi_backend.bitcask_1day.bitcask.expiry = 1d
    > multi_backend.bitcask_1day.bitcask.expiry.grace_time = 1h
    > multi_backend.bitcask_2day.storage_backend = bitcask
    > multi_backend.bitcask_2day.bitcask.data_root = /var/lib/riak/bitcask_2day
    > multi_backend.bitcask_2day.bitcask.expiry = 2d
    > multi_backend.bitcask_2day.bitcask.expiry.grace_time = 1h
    > multi_backend.bitcask_4day.storage_backend = bitcask
    > multi_backend.bitcask_4day.bitcask.data_root = /var/lib/riak/bitcask_4day
    > multi_backend.bitcask_4day.bitcask.expiry = 4d
    > multi_backend.bitcask_4day.bitcask.expiry.grace_time = 1h
    > multi_backend.bitcask_8day.storage_backend = bitcask
    > multi_backend.bitcask_8day.bitcask.data_root = /var/lib/riak/bitcask_8day
    > multi_backend.bitcask_8day.bitcask.expiry = 8d
    > multi_backend.bitcask_8day.bitcask.expiry.grace_time = 1h
    > multi_backend.leveldb_mult.storage_backend = leveldb
    > multi_backend.leveldb_mult.leveldb.maximum_memory.percent = 30
    > multi_backend.leveldb_mult.leveldb.data_root = /var/lib/riak/leveldb_mult
    > multi_backend.bitcask_mult.storage_backend = bitcask
    > multi_backend.bitcask_mult.bitcask.data_root = /var/lib/riak/bitcask_mult
    > multi_backend.default = leveldb_mult
    > 
    > $ du –h –d 1 /var/lib/riak
    > 
    > 4.0K    /var/lib/riak/riak_kv_exchange_fsm
    > 52K     /var/lib/riak/generated.configs
    > 224K    /var/lib/riak/cluster_meta
    > 424K    /var/lib/riak/ring
    > 1.1M    /var/lib/riak/kv_vnode
    > 2.1M    /var/lib/riak/bitcask_1day
    > 3.6M    /var/lib/riak/bitcask_4day
    > 33M     /var/lib/riak/yz_temp
    > 1.1G    /var/lib/riak/bitcask_2day
    > 5.9G    /var/lib/riak/yz_anti_entropy
    > 20G     /var/lib/riak/yz
    > 24G     /var/lib/riak/leveldb_mult
    > 25G     /var/lib/riak/bitcask_mult
    > 27G     /var/lib/riak/bitcask_8day
    > 139G    /var/lib/riak/anti_entropy
    > 240G    /var/lib/riak
    > 
    > 
    > -- 
    > Rohit S.
    > 
    > _______________________________________________
    > riak-users mailing list
    > riak-users at lists.basho.com
    > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.basho.com_mailman_listinfo_riak-2Dusers-5Flists.basho.com&d=DwIFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=qO7g3ZLnCDJ2Kp8qz13HpzC2NAX5qNCtEhoWeGshuEI&m=zcfqvyFJVK17vAft30Cv9abkK5P_f2JtRonrl0CRYnM&s=wSNoX28DcQ-CCfWp9vRUA-70z3LeWFNGHG3-LPKzAtQ&e= 
    
    



More information about the riak-users mailing list