RiakSearch Reached Its Limit and gave below Error ..

Rusty Klophaus rusty at basho.com
Sun Oct 31 11:19:17 EDT 2010


Hi Prometheus,

This is a known issue, tracked here (https://issues.basho.com/790) and high
on the list for the next release.

The summary is that when putting data into Riak Search too quickly, it will
build up a backlog of files waiting to be compacted. Each file maintains its
own separate ETS table, and this causes the system to run out of tables.

You can quickly check how many files Riak Search has open using "find
./data/merge_index | wc -l". Ideally, during indexing there should be <15
per partition. If this grows much beyond that, it's a sign to slow down
indexing.

Best,
Rusty

On Sun, Oct 31, 2010 at 6:13 AM, Prometheus WillSurvive <
prometheus.willsurvive at gmail.com> wrote:

> I put the ERL_MAX_ETS_TABLES  in vm.args  and restart riaksearch  its ok
> now..
>
> thanks..
>
>
>
> On Oct 31, 2010, at 11:38 AM, Ulf Wiger wrote:
>
> >
> > It looks as if it is the ets:new/2 operation that fails due to
> system_limit.
> >
> > Like with the number of concurrently running processes, the system limit
> > is by default pretty low (1400 tables), but can be increased by setting
> the
> > OS environment variable ERL_MAX_ETS_TABLES.
> >
> > Now, I have no idea if Riak Search is supposed to need tons of ets
> > tables. I know of situations where reaching the ets system limit can be a
> > symptom of growing backlog in the system - e.g. if mnesia is falling
> behind,
> > since every pending mnesia transaction keeps at least one ets table for
> > the temporary transaction store.
> >
> > BR,
> > Ulf W
> >
> > On 31 Oct 2010, at 09:23, Prometheus WillSurvive wrote:
> >
> >> Hi,
> >>
> >> We started  a batch index test (wikipedia)  when we reached around 600K
> docs  system gave below error..  Any idea ?
> >>
> >> We can not index any more doc in this index.
> >>
> >>
> >>
> >> =ERROR REPORT==== 31-Oct-2010::10:22:42 ===
> >> ** Too many db tables **
> >>
> >> DEBUG: riak_search_dir_indexer:197 - "{ error , Type , Error , erlang :
> get_stacktrace ( ) }"
> >>
> >> {error,error,system_limit,
> >>       [{ets,new,[batch,[protected,duplicate_bag]]},
> >>        {riak_search_client,process_terms_1,5},
> >>        {riak_search_client,process_terms_1,5},
> >>        {riak_search_client,index_docs,3},
> >>        {riak_search_client,index_docs,2},
> >>        {riak_solr_search_client,run_solr_command,4},
> >>        {solr_search,'-index_dir/2-lc$^0/1-0-',2},
> >>        {riak_search_dir_indexer,worker_loop,5}]}
> >>
> >>
> >> =ERROR REPORT==== 31-Oct-2010::10:22:44 ===
> >> Error in process <0.22187.14> on node 'riaksearch at 192.168.250.154' with
> exit value: {system_limit,[{riak_search_dir_indexer,worker_loop,5}]}
> >>
> >> RPC to 'riaksearch at 192.168.250.154' failed: {'EXIT',
> >>                                            {system_limit,
> >>                                             [{riak_search_dir_indexer,
> >>                                               worker_loop,5}]}}
> >>
> >>
> >> _______________________________________________
> >> riak-users mailing list
> >> riak-users at lists.basho.com
> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
> > Ulf Wiger, CTO, Erlang Solutions, Ltd.
> > http://erlang-solutions.com
> >
> >
> >
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20101031/2cd93ff3/attachment.html>


More information about the riak-users mailing list