General Memory/Performance Tuning Guidelines
gtillman at mezeo.com
Wed May 25 08:29:48 EDT 2011
Thanks very much for your time and trouble. Great info, very helpful, and very timely!
On May 24, 2011, at 16:32 , Rusty Klophaus wrote:
I have limited knowledge of configuring Innostore but can help answer some of your merge_index questions.
The most important merge_index setting in terms of memory usage is 'buffer_rollover_size'. This affects how large the buffer is allowed to grow, in bytes, before getting converted to an on-disk segment. Each partition maintains a separate buffer, so any increases to this number will be multiplied by the number of partitions in your system. The higher this number, the less frequently merge_index will need to perform compactions.
The second most important settings for memory usage are a combination of 'segment_full_read_size' and 'max_compact_segments'. During compaction, the system will completely page any segments smaller than the 'segment_full_read_size' value into memory. This should generally be as large or larger than the 'buffer_rollover_size'. The higher this number, the quicker each compaction will be. 'max_compact_segments' is the maximum number of segments to compact at one time. The higher this number, the more segments merge_index can involve in each compaction. In the worst case, a compaction could take ('segment_full_read_size' * 'max_compact_segments') bytes of RAM.
The rest of the settings have a much smaller impact on performance and memory usage, and exist mainly for tweaking and special cases.
This is a completely unscientific estimate based on observing other Riak Search applications, but I'd set buffer_rollover_size so that (# Partitions * buffer_rollover_size) is about one-half the memory you wish for merge_index to consume, hopefully somewhere between 1M and 10M. The rest of the memory will be used by in-memory offset tables, compaction processes, and during query operations.
Hope that helps.
On Mon, May 23, 2011 at 2:05 PM, Gordon Tillman <gtillman at mezeo.com<mailto:gtillman at mezeo.com>> wrote:
We are working with a riaksearch cluster that uses innostore as the primary backend in tandem with merge_index that is required by search. From reading the Basho wiki it looks like the following are the most important factors affecting memory and performance:
• put data_home_dir and log_group_home_dir on different spindles
Ideally, data_home_dir, log_group_home_dir, and data_root would all be on different spindles, but if you had just 2 disks available what would you recommend? Would it be best to have data_home_dir and data_root on one and then log_group_home_dir on the other?
in calculating the proper setting for buffer_pool_size you are directed to allocate 60-80 percent of available RAM. So lets assume you want to take the remaining 20-40% of available RAM and split it up between innostore and merge_index?
Would it be best to give each of them half of that value?
Determining the approximate memory requirements for merge_index isn't (to me) real obvious. I looks like the following all have an effect:
Is there a formula for determining the (approximate) proper values to use given a certain amount of available RAM?
Thanks in advance for any advice. Sorry for all the questions!
riak-users mailing list
riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the riak-users