<div dir="ltr">Hi<div>No I'm using bitcask</div><div><br></div><div>basho_ bench configurations are:</div><div><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">{mode, {rate, 300}}.

{duration, 4320}.

{report_interval, 1}.

{concurrent, 15}.

{driver, basho_bench_driver_riakc_pb}.

{key_generator, {int_to_bin_bigendian, {uniform_int, 1000000000}}}.

{value_generator, {fixed_bin, 1000}}.

{riakc_pb_ips, [{192,168,1,110},{192,168,1,111},{192,168,1,112}]}.

{riakc_pb_replies, 1}.

{riakc_pb_bucket, {<<"n_val_of_2">>,<<"bin-test">>}}.

{operations, [{put, 1}]}.

%% Use {auto_reconnect, false} to get "old" behavior (prior to April 2013).
%% See deps/riakc/src/riakc_pb_socket.erl for all valid socket options.
{pb_connect_options, [{auto_reconnect, true}]}.

%% Overrides for the PB client's default 60 second timeout, on a
%% per-type-of-operation basis.  All timeout units are specified in
%% milliseconds.  The pb_timeout_general config item provides a
%% default timeout if the read/write/listkeys/mapreduce timeout is not
%% specified.

{pb_timeout_general, 30000}.
{pb_timeout_read, 5000}.
{pb_timeout_write, 50000}.
{pb_timeout_listkeys, 50000}.
%% The general timeout will be used because this specific item is commented:
%% {pb_timeout_mapreduce, 50000}.</pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">My OS and riak ocnfig:</pre><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap"><pre style="word-wrap:break-word;white-space:pre-wrap">cat /sys/kernel/mm/transparent_hugepage/enabled :
=================================================
always madvise [never]


cat /etc/fstab:
===============
/dev/sdb1       /mnt/storage    ext4    defaults,noatime,barrier=0,data=writeback      0      1


cat /sys/block/sda/queue/scheduler:
===================================
noop [deadline] cfq


cat /sys/block/sda/queue/nr_requests:
=====================================
128


cat /etc/sysctl.conf:
=====================
vm.swappiness=0
vm.dirty_background_ratio=0
vm.dirty_background_bytes=209715200
vm.dirty_ratio=0
vm.dirty_bytes=2000000000
vm.dirty_writeback_centisecs=100
vm.dirty_expire_centisecs=200

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

net.core.wmem_max=16777216
net.core.rmem_max=16777216
net.ipv4.tcp_mem=16777216 16777216 16777216
net.ipv4.tcp_rmem=10240 87380 16777216
net.ipv4.tcp_wmem=10240 87380 16777216
net.core.netdev_max_backlog=500000

net.core.rmem_default=16777216
net.core.wmem_default=16777216

net.ipv4.tcp_max_syn_backlog = 500000
net.core.somaxconn = 500000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_syncookies = 1

net.ipv4.ip_local_port_range=1024 655350
fs.file-max=10000000


cat /etc/riak/riak.conf:
========================
log.console = file
log.console.level = info
log.console.file = $(platform_log_dir)/console.log
log.error.file = $(platform_log_dir)/error.log
log.syslog = off
log.crash = on
log.crash.file = $(platform_log_dir)/crash.log
log.crash.maximum_message_size = 64KB
log.crash.size = 10MB
log.crash.rotation = $D0
log.crash.rotation.keep = 5
nodename = <a href="mailto:riak@192.168.1.111">riak@192.168.1.111</a>
distributed_cookie = riak
erlang.async_threads = 1000
erlang.max_ports = 262144
erlang.smp = auto
erlang.distribution_buffer_size = 2097151KB
erlang.async_threads.stack_size = 5MB
erlang.process_limit = 10000000
max_concurrent_requests = 3000000
background_manager = on
ring_size = 64
dtrace = off
platform_bin_dir = /usr/sbin
platform_data_dir = /var/lib/riak
platform_etc_dir = /etc/riak
platform_lib_dir = /usr/lib/riak/lib
platform_log_dir = /var/log/riak
listener.http.internal = <a href="http://0.0.0.0:8098">0.0.0.0:8098</a>
listener.protobuf.internal = <a href="http://192.168.1.111:8087">192.168.1.111:8087</a>
protobuf.backlog = 102400
anti_entropy = active
storage_backend = bitcask
object.format = 1
object.size.warning_threshold = 10MB
object.size.maximum = 50MB
object.siblings.warning_threshold = 25
object.siblings.maximum = 100
bitcask.data_root = /mnt/storage/bitcask
bitcask.io_mode = erlang
riak_control = off
riak_control.auth.mode = off
leveldb.maximum_memory.percent = 90
leveldb.compression = on
leveldb.compression.algorithm = lz4
search = off
search.solr.start_timeout = 30s
search.solr.port = 8093
search.solr.jmx_port = 8985
search.solr.jvm_options = -d64 -Xms1g -Xmx1g -XX:+UseStringCache -XX:+UseCompressedOops
bitcask.max_file_size = 10GB
</pre><div>and my results are attached to mail.</div><div>thanks</div></pre></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 19, 2017 at 7:32 PM, Travis Kirstine <span dir="ltr"><<a href="mailto:tkirstine@firstbasesolutions.com" target="_blank">tkirstine@firstbasesolutions.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">





<div lang="EN-CA" link="#0563C1" vlink="#954F72">
<div class="m_6559969529571998237WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Are you using leveldb?   Matthew (Basho) talk “Optimizing LevelDB for Performance and Scale” available in the link below provides a
 great overview<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><a href="https://www.youtube.com/watch?v=vo88IdglU_8&t=1437s" target="_blank">https://www.youtube.com/watch?<wbr>v=vo88IdglU_8&t=1437s</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">The talk dates from 2013 and some things have changed but I found it very helpful<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Have you gone through
<a href="http://docs.basho.com/riak/kv/2.2.3/using/performance/" target="_blank">http://docs.basho.com/riak/kv/<wbr>2.2.3/using/performance/</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Besides that you’d need to provide more details on your setup<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Regards<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> riak-users [mailto:<a href="mailto:riak-users-bounces@lists.basho.com" target="_blank">riak-users-bounces@<wbr>lists.basho.com</a>]
<b>On Behalf Of </b>mohammad sharifi<br>
<b>Sent:</b> June-19-17 8:48 AM<br>
<b>To:</b> <a href="mailto:riak-users@lists.basho.com" target="_blank">riak-users@lists.basho.com</a><br>
<b>Subject:</b> Riak overall performance falls down<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">I have a clustered riak node, when I'm using basho bench to take a benchmark, performance falls down, (in spite of for example, 1K or 10K insert rate) after some time, overall insert rate is not smooth at all.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">OS status after performance reduction: High memory usage, high user I/O wait<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">is there any one to explain why riak behaving like this?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">thanks in advance<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal">Mohammad Sharifi<br>
Real Eyes Realize Real Lies<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div></div></div>
</div>

</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Mohammad Sharifi<br>Real Eyes Realize Real Lies</div>
</div>