riak + innostore

David Smith dizzyd at basho.com
Fri Feb 19 11:57:24 EST 2010


Hi Lev,

On Fri, Feb 19, 2010 at 2:03 AM, Lev Walkin <vlm at lionet.info> wrote:

>
> First, we noticed that innostore was slow accepting data:
>
>  timer:tc(innostore_riak, put, [S1, {<<"siden">>, <<"key">>},
>> <<"value">>]).
>> {8995645,ok}
>> timer:tc(innostore_riak, put, [S1, {<<"siden">>, <<"key">>},
>> <<"value">>]).
>> {4834159,ok}
>>
>
On my own box, I verified this odd behaviour:

Orig:
timer:tc(innostore_riak, put, [S1, {<<"siden">>, <<"key">>}, <<"value">>]).
{28429,ok}
timer:tc(innostore_riak, put, [S1, {<<"siden">>, <<"key">>}, <<"value">>]).
{927,ok}

Patched:
timer:tc(innostore_riak, put, [S1, {<<"siden">>, <<"key">>}, <<"value">>]).
{4237,ok}
timer:tc(innostore_riak, put, [S1, {<<"siden">>, <<"key">>}, <<"value">>]).
{726,ok}

Now, while I agree that the patched version is faster for the first pass of
this particular micro-benchmark, the second pass of the test is close enough
that I'm not convinced that port_command is significantly better. In
general, I'm skeptical of single-data point benchmarking; it's just too easy
to run into red-herrings. Did you run your own before/after test (beyond
console testing) to verify your fix?

As a cross-check, I ran a 15-minute benchmark against the two versions of
inno -- they are attached as inno_orig and inno_patched, respectively. As
you can see, inno_orig has roughly same (and slightly better throughput)
than inno_patched. However, the latencies are where things get interesting.
inno_orig has an mean 95th percentile latency of 5.4 ms on GET, compared to
inno_patch's 8.3 ms. The mean 95th percentile latency for PUT is even more
telling: 18.0 ms for inno_orig vs. 26.1 ms for inno_patched.

It's also worth noting that the time required for the mean/median/95th
percentiles to converge on GET operations takes ~30 seconds longer on the
inno_patched; this is the amount of time required to load the entire dataset
in memory -- thus confirming that the total throughput is higher on
inno_org.

port_control is supposed to be the fastest way, per the OTP team, to get
in/out of the emulator to a port driver. Now, given that the innostore
driver sends messages back, we may be losing the bulk of the speed advantage
of port_control -- this is my best guess as to why we see these differences.
However, as I demonstrated above, the difference in a large-scale test
is negligible and port_control still comes out on top (if only from a
latency standpoint).

I would also note that if you're going to do micro-benchmarks, please don't
do it on EC2 -- esp. the small instances. As I'm sure you know,
micro-benchmarks are heavily influenced by the environment and virtualized
platforms just introduce too much jitter to yield repeatable results. There
is also a growing amount of anecdotal evidence that EC2 small instances, in
particular, suffer from this problem.

After doing the patch, we've found an order of magnitude difference between
> calling innostore directly and using it as a riak backend.
>

Comparing raw calls to innostore with riak client is an invalid comparison.
Riak adds a number of processing steps to every operation to provide the
eventually consistent distributed storage that we all know and love. In
addition, the amount of data that is actually getting stored for every Riak
request is quite a bit more, maybe even 10x the size of the original data
(which is so small that 10x is really not that much bigger).

Again, I would caution against this sort of micro-benchmarking -- it's not
at all representative of the performance you will see in a production
environment. As the data set on a production cluster grows, you quickly
become bound not by the speed of Riak, but by the I/O constraints on each
box. Micro-benchmarks such as the ones you use here often lead to premature
(and unnecessary) optimization. The valid window for these types of
benchmarks is the first 5 minutes of runtime in your system, before you have
any significant data in the system and everything still fits into cache.

All of my ranting about micro-benchmarks aside, you did uncover something
interesting about the behaviour of port_control vs port_command. I'm not
sure how to explain it and if you have additional evidence, I'll gladly
review it.

D.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20100219/3ed4fab1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: inno_patched.png
Type: image/png
Size: 128957 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20100219/3ed4fab1/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: inno_orig.png
Type: image/png
Size: 128924 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20100219/3ed4fab1/attachment-0001.png>


More information about the riak-users mailing list