Riak overall performance falls down

mohammad sharifi mohammadsharifi666 at gmail.com
Tue Jun 20 04:56:22 EDT 2017


Hi
No I'm using bitcask

basho_ bench configurations are:

{mode, {rate, 300}}.

{duration, 4320}.

{report_interval, 1}.

{concurrent, 15}.

{driver, basho_bench_driver_riakc_pb}.

{key_generator, {int_to_bin_bigendian, {uniform_int, 1000000000}}}.

{value_generator, {fixed_bin, 1000}}.

{riakc_pb_ips, [{192,168,1,110},{192,168,1,111},{192,168,1,112}]}.

{riakc_pb_replies, 1}.

{riakc_pb_bucket, {<<"n_val_of_2">>,<<"bin-test">>}}.

{operations, [{put, 1}]}.

%% Use {auto_reconnect, false} to get "old" behavior (prior to April 2013).
%% See deps/riakc/src/riakc_pb_socket.erl for all valid socket options.
{pb_connect_options, [{auto_reconnect, true}]}.

%% Overrides for the PB client's default 60 second timeout, on a
%% per-type-of-operation basis.  All timeout units are specified in
%% milliseconds.  The pb_timeout_general config item provides a
%% default timeout if the read/write/listkeys/mapreduce timeout is not
%% specified.

{pb_timeout_general, 30000}.
{pb_timeout_read, 5000}.
{pb_timeout_write, 50000}.
{pb_timeout_listkeys, 50000}.
%% The general timeout will be used because this specific item is commented:
%% {pb_timeout_mapreduce, 50000}.

My OS and riak ocnfig:

cat /sys/kernel/mm/transparent_hugepage/enabled :
=================================================
always madvise [never]


cat /etc/fstab:
===============
/dev/sdb1       /mnt/storage    ext4
defaults,noatime,barrier=0,data=writeback      0      1


cat /sys/block/sda/queue/scheduler:
===================================
noop [deadline] cfq


cat /sys/block/sda/queue/nr_requests:
=====================================
128


cat /etc/sysctl.conf:
=====================
vm.swappiness=0
vm.dirty_background_ratio=0
vm.dirty_background_bytes=209715200
vm.dirty_ratio=0
vm.dirty_bytes=2000000000
vm.dirty_writeback_centisecs=100
vm.dirty_expire_centisecs=200

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

net.core.wmem_max=16777216
net.core.rmem_max=16777216
net.ipv4.tcp_mem=16777216 16777216 16777216
net.ipv4.tcp_rmem=10240 87380 16777216
net.ipv4.tcp_wmem=10240 87380 16777216
net.core.netdev_max_backlog=500000

net.core.rmem_default=16777216
net.core.wmem_default=16777216

net.ipv4.tcp_max_syn_backlog = 500000
net.core.somaxconn = 500000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_syncookies = 1

net.ipv4.ip_local_port_range=1024 655350
fs.file-max=10000000


cat /etc/riak/riak.conf:
========================
log.console = file
log.console.level = info
log.console.file = $(platform_log_dir)/console.log
log.error.file = $(platform_log_dir)/error.log
log.syslog = off
log.crash = on
log.crash.file = $(platform_log_dir)/crash.log
log.crash.maximum_message_size = 64KB
log.crash.size = 10MB
log.crash.rotation = $D0
log.crash.rotation.keep = 5
nodename = riak at 192.168.1.111
distributed_cookie = riak
erlang.async_threads = 1000
erlang.max_ports = 262144
erlang.smp = auto
erlang.distribution_buffer_size = 2097151KB
erlang.async_threads.stack_size = 5MB
erlang.process_limit = 10000000
max_concurrent_requests = 3000000
background_manager = on
ring_size = 64
dtrace = off
platform_bin_dir = /usr/sbin
platform_data_dir = /var/lib/riak
platform_etc_dir = /etc/riak
platform_lib_dir = /usr/lib/riak/lib
platform_log_dir = /var/log/riak
listener.http.internal = 0.0.0.0:8098
listener.protobuf.internal = 192.168.1.111:8087
protobuf.backlog = 102400
anti_entropy = active
storage_backend = bitcask
object.format = 1
object.size.warning_threshold = 10MB
object.size.maximum = 50MB
object.siblings.warning_threshold = 25
object.siblings.maximum = 100
bitcask.data_root = /mnt/storage/bitcask
bitcask.io_mode = erlang
riak_control = off
riak_control.auth.mode = off
leveldb.maximum_memory.percent = 90
leveldb.compression = on
leveldb.compression.algorithm = lz4
search = off
search.solr.start_timeout = 30s
search.solr.port = 8093
search.solr.jmx_port = 8985
search.solr.jvm_options = -d64 -Xms1g -Xmx1g -XX:+UseStringCache
-XX:+UseCompressedOops
bitcask.max_file_size = 10GB

and my results are attached to mail.
thanks


On Mon, Jun 19, 2017 at 7:32 PM, Travis Kirstine <
tkirstine at firstbasesolutions.com> wrote:

> Are you using leveldb?   Matthew (Basho) talk “Optimizing LevelDB for
> Performance and Scale” available in the link below provides a great overview
>
>
>
> https://www.youtube.com/watch?v=vo88IdglU_8&t=1437s
>
>
>
> The talk dates from 2013 and some things have changed but I found it very
> helpful
>
>
>
> Have you gone through http://docs.basho.com/riak/kv/
> 2.2.3/using/performance/
>
>
>
> Besides that you’d need to provide more details on your setup
>
>
>
> Regards
>
>
>
> *From:* riak-users [mailto:riak-users-bounces at lists.basho.com] *On Behalf
> Of *mohammad sharifi
> *Sent:* June-19-17 8:48 AM
> *To:* riak-users at lists.basho.com
> *Subject:* Riak overall performance falls down
>
>
>
> Hi
>
> I have a clustered riak node, when I'm using basho bench to take a
> benchmark, performance falls down, (in spite of for example, 1K or 10K
> insert rate) after some time, overall insert rate is not smooth at all.
>
> OS status after performance reduction: High memory usage, high user I/O
> wait
>
> is there any one to explain why riak behaving like this?
>
> thanks in advance
>
>
>
>
> --
>
> Mohammad Sharifi
> Real Eyes Realize Real Lies
>



-- 
Mohammad Sharifi
Real Eyes Realize Real Lies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170620/c86b726f/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: summary-12.png
Type: image/png
Size: 93655 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170620/c86b726f/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: summary-13.png
Type: image/png
Size: 85908 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170620/c86b726f/attachment-0001.png>


More information about the riak-users mailing list