Performance Tuning in OmniOS

Jared Morrow jared at basho.com
Tue Jan 21 15:51:19 EST 2014


Oh I think OmniOS is far from hopeless.  The problem you are having is the
same problem you'd have if you were on ubuntu and you made a LVM raid on
vanilla EBS.  EBS is the problem when it comes to predictable write / read
speed.  People still use it, but not without careful thought and
consideration.  You can try using provisioned IOPS for EBS, which the
m1.large supports, or ask in risk-users what other AWS users have setup.  I
know we have a lot of customers and OSS users running on AWS, so they are
far more knowledgeable about real-world performance than I am.

Good luck,
Jared




On Tue, Jan 21, 2014 at 12:05 PM, Hari John Kuriakose <ejhari at gmail.com>wrote:

> I am using the default raid itself.
>
> Well, if this is the case, I will run the tests again with a different
> setup as you said, and get back as soon as possible. I would just like to
> believe that OmniOS is not too hopeless.
>
> Thank you.
> On Jan 21, 2014 11:17 PM, "Jared Morrow" <jared at basho.com> wrote:
>
>> What type of RAID did you chose for your spool of 5 volumes?  If you
>> chose the default of raidz, you will not be getting much of a performance
>> boost over vanilla EBS, just a big integrity boost.  Also, unless you are
>> using provisioned IOPS for EBS, you are starting from an extremely slow
>> base-case, so adding ZFS on top might not help matters much.
>>
>> If speed is the concern, as a test I'm willing to bet if you do another
>> test run against the two instance storage disks on that m1.large, you will
>> probably beat those 5 EBS volumes pretty easily.
>>
>> -Jared
>>
>>
>> On Tue, Jan 21, 2014 at 9:22 AM, Hari John Kuriakose <ejhari at gmail.com>wrote:
>>
>>> Hello,
>>>
>>> I am using standard EBS devices, with a zpool in an instance comprising
>>> of five 40GB volumes.
>>> Each of the Riak instance is of m1.large type.
>>>
>>> I have made the following changes in zfs properties:
>>>
>>> # My reason: the default sst block size for leveldb is 4k.
>>> zfs set recordsize=4k tank/riak
>>> # My reason: by default, leveldb verifies checksums automatically.
>>> zfs set checksum=off tank/riak
>>> zfs set atime=off tank/riak
>>> zfs set snapdir=visible tank/riak
>>>
>>> And I did the following with help from Basho AWS tuning docs:
>>>
>>> projadd -c "riak" -K "process.max-file-descriptor=(basic,65536,deny)"
>>> user.riak
>>> bash -c "echo 'set rlim_fd_max=65536' >> /etc/system"
>>> bash -c "echo 'set rlim_fd_cur=65536' >> /etc/system"
>>> ndd -set /dev/tcp tcp_conn_req_max_q0 40000
>>> ndd -set /dev/tcp tcp_conn_req_max_q 4000
>>> ndd -set /dev/tcp tcp_tstamp_always 0
>>> ndd -set /dev/tcp tcp_sack_permitted 2
>>> ndd -set /dev/tcp tcp_wscale_always 1
>>> ndd -set /dev/tcp tcp_time_wait_interval 60000
>>> ndd -set /dev/tcp tcp_keepalive_interval 120000
>>> ndd -set /dev/tcp tcp_xmit_hiwat 2097152
>>> ndd -set /dev/tcp tcp_recv_hiwat 2097152
>>> ndd -set /dev/tcp tcp_max_buf 8388608
>>>
>>> Thanks again.
>>>
>>>
>>> On Tue, Jan 21, 2014 at 9:12 PM, Hector Castro <hector at basho.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> Can you please clarify what type of disk you are using within AWS?
>>>> EBS, EBS with PIOPS, instance storage? In addition, maybe some details
>>>> on volume sizes and instance types.
>>>>
>>>> These details may help someone attempting to answer your question.
>>>>
>>>> --
>>>> Hector
>>>>
>>>>
>>>> On Tue, Jan 21, 2014 at 8:11 AM, Hari John Kuriakose <ejhari at gmail.com>
>>>> wrote:
>>>> >
>>>> > I am running LevelDB on ZFS in Solaris (OmniOS specifically) in
>>>> Amazon AWS.
>>>> > The iops is very very low. There is no significant progress with
>>>> tuning too.
>>>> >
>>>> > Why I chose ZFS is that since LevelDB requires the node to be stopped
>>>> before
>>>> > taking a backup, I needed a filesystem with snapshot ability. And the
>>>> most
>>>> > favourable Amazon community AMI seemed to be using OmniOS (fork of
>>>> Solaris).
>>>> > Everything is fine, except the performance.
>>>> >
>>>> > I did all the AWS tuning proposed by Basho but still Basho Bench gave
>>>> twice
>>>> > iops on Ubuntu as compared to OmniOS, under same conditions. Also, I
>>>> am
>>>> > using riak-js client library, and its a 5 node Riak cluster with 8GB
>>>> ram
>>>> > each.
>>>> >
>>>> > Could not yet figure out what is really causing the congestion in
>>>> OmniOS.
>>>> > Any pointers will be really helpful.
>>>> >
>>>> > Thanks and regards,
>>>> > Hari John Kuriakose.
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > riak-users mailing list
>>>> > riak-users at lists.basho.com
>>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>> >
>>>>
>>>
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140121/a953dd15/attachment.html>


More information about the riak-users mailing list