eLevelDB max_open_files in 1.2.0

Mark Phillips mark at basho.com
Fri Aug 17 10:12:44 EDT 2012


Hi Dave,

On Thu, Aug 16, 2012 at 9:41 AM, Dave Brady <dbrady at weborama.com> wrote:
> Thanks, Mark!
>
> I have a somewhat-related follow up question: is there a recommended maximum for the amount of data held on a machine?
>
> I ask because at my previous company Cassandra was used, and we were advised to put no more 250 GB per physical box.  The reasoning was that in a failure situation, rebuilding any more than this amount of data would cause too great a performance degradation.
>

So there's no hard and fast amount of data per node that you shouldn't
be passing when using Riak. Rather, it makes more sense to keep in
mind the resources needed when a node (or set of nodes) becomes
unreachable, dead, or is sold on ebay by a sysadmin trying to raise
money for a ping pong table..  Plan appropriately for capacity and
Riak should operate as advertised . 250GB/node shouldn't be an issue
as long as you understand the resources needed to keep the cluster
going in failure and growth scenarios. (Anecdotally, I can tell you I
know of one Riak user that has >5TB/physical node on a cluster with
10s of nodes.)

Ryan Zezeski also pointed out that Eric Brewer wrote a paper in 2001
[0] that touches on this and is worth a read (much like all the papers
he writes).

Hope that helps.

Mark
ricon2012.com

[0] http://www.cs.berkeley.edu/~brewer/Giant.pdf


> --
> Dave Brady
>
> ----- Original Message -----
> From: "Mark Phillips" <mark at basho.com>
> To: "Dave Brady" <dbrady at weborama.com>
> Cc: riak-users at lists.basho.com
> Sent: Tuesday, August 14, 2012 7:34:10 PM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna
> Subject: Re: eLevelDB max_open_files in 1.2.0
>
> Hi Dave,
>
> On Sun, Aug 12, 2012 at 3:58 PM, Dave Brady <dbrady at weborama.com> wrote:
>> Hello,
>>
>> First I want to thanks Basho for greatly expanding the documentation on the Wiki for configuring/tuning Riak and eLevelDB in 1.2.0!  Big improvement over 1.1.x.
>>
>> My question is about max_open_files: here the documentation is confusing to me.
>>
>> It says to allocate one open file per 2 MB, then divide by the number of partitions.  This is the same formula used in 1.1.x.
>>
>> It goes on to say that if you manually set this parameter in 1.1.x, to divide that value by two for 1.2.0.
>>
>> Should not the formula for 1.2.0, in that case, read as use one file per 4 MB?
>
> Long story short, the answer is "yes" :)
>
> With 1.2, 4MB is the advised file size, and you should be running with
> no less that 20 files/nodes. I'll take a pass at updating the docs to
> make this a bit easier to understand. Thanks for pointing that out.
>
> Mark
>
>>
>> Thankks!
>>
>> --
>> Dave Brady
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




More information about the riak-users mailing list