File descriptor leaks?

Dmitry Demeshchuk demeshchuk at gmail.com
Mon Oct 18 23:07:48 EDT 2010


Hi, Justin.

We are using 0.12.1.

I thought it could be caused by our version of OTP (we had been using
R14A) so I downgraded to R13B04. It didn't help though.

Unfortunately, there's nothing interesting in logs, neither in Riak's
nor in ours. I think that most likely there are some web requests that
hang for some reason. Moreover, since we haven't been getting this
problem before the last week I guess there's some problem in my own
code that I have released recently.

However, for now I got some more urgent tasks to finish so I just
added a cron job that restarts Riak every hour. Guess I'll have to
handle this problem in a week or so on.

Anyway, thanks for getting in touch. As soon as I have time to
investigate the problem I'll let you know the result.

On Mon, Oct 18, 2010 at 6:11 PM, Justin Sheehy <justin at basho.com> wrote:
> Hi, Dmitry.
>
> What version of Riak are you using?  And is there anything interesting
> in the error logs?
>
> -Justin
>
>
>
>
> On Thu, Oct 14, 2010 at 7:53 AM, Dmitry Demeshchuk <demeshchuk at gmail.com> wrote:
>> A small update. I've just encountered the same problem. Just about 3-4
>> hours have passed.
>>
>> lsof | wc -l showed only about 2k descriptors for all users. That's
>> even more weird as the 32k descriptors limit is per user. So, we
>> haven't reached the limit so far.
>>
>> On Thu, Oct 14, 2010 at 3:48 PM, Dmitry Demeshchuk <demeshchuk at gmail.com> wrote:
>>> Greetings.
>>>
>>> We have recently started to get the emfile errors. ulimit -n is 32767.
>>> Restarting Riak helps for several hours and then we run out of
>>> descriptors again.
>>>
>>> Some time later after restart I performed lsof and found the following
>>> descriptors:
>>>
>>> kondemand   154       root  cwd   unknown
>>> /proc/154/cwd (readlink: Permission denied)
>>> kondemand   154       root  rtd   unknown
>>> /proc/154/root (readlink: Permission denied)
>>> kondemand   154       root  txt   unknown
>>> /proc/154/exe (readlink: Permission denied)
>>> kondemand   154       root NOFD
>>> /proc/154/fd (opendir: Permission denied)
>>> kondemand   155       root  cwd   unknown
>>> /proc/155/cwd (readlink: Permission denied)
>>> kondemand   154       root  cwd   unknown
>>> /proc/154/cwd (readlink: Permission denied)
>>> kondemand   154       root  rtd   unknown
>>> /proc/154/root (readlink: Permission denied)
>>> kondemand   154       root  txt   unknown
>>> /proc/154/exe (readlink: Permission denied)
>>> kondemand   154       root NOFD
>>> /proc/154/fd (opendir: Permission denied)
>>> kondemand   155       root  cwd   unknown
>>> /proc/155/cwd (readlink: Permission denied)
>>> kondemand   155       root  rtd   unknown
>>> /proc/155/root (readlink: Permission denied)
>>> kondemand   155       root  txt   unknown
>>> /proc/155/exe (readlink: Permission denied)
>>> kondemand   155       root NOFD
>>> /proc/155/fd (opendir: Permission denied)
>>> kondemand   156       root  cwd   unknown
>>> /proc/156/cwd (readlink: Permission denied)
>>> kondemand   156       root  rtd   unknown
>>> /proc/156/root (readlink: Permission denied)
>>> kondemand   156       root  txt   unknown
>>> /proc/156/exe (readlink: Permission denied)
>>> kondemand   156       root NOFD
>>> /proc/156/fd (opendir: Permission denied)
>>> kondemand   157       root  cwd   unknown
>>> /proc/157/cwd (readlink: Permission denied)
>>> kondemand   157       root  rtd   unknown
>>> /proc/157/root (readlink: Permission denied)
>>> kondemand   157       root  txt   unknown
>>> /proc/157/exe (readlink: Permission denied)
>>> kondemand   157       root NOFD
>>> /proc/157/fd (opendir: Permission denied)
>>> kondemand   158       root  cwd   unknown
>>> /proc/158/cwd (readlink: Permission denied)
>>> kondemand   158       root  rtd   unknown
>>> /proc/158/root (readlink: Permission denied)
>>> kondemand   158       root  txt   unknown
>>> /proc/158/exe (readlink: Permission denied)
>>>
>>> Also, the following couple of descriptors is opened several times at
>>> the same time:
>>>
>>> bash      20176        dem  mem       REG     252,0   256316   1179925
>>> /usr/lib/locale/en_US.utf8/LC_CTYPE
>>> bash      20176        dem  mem       REG     252,0       54   1179926
>>> /usr/lib/locale/en_US.utf8/LC_NUMERIC
>>> bash      20176        dem  mem       REG     252,0     2454   1179927
>>> /usr/lib/locale/en_US.utf8/LC_TIME
>>> bash      20176        dem  mem       REG     252,0   966938   1179928
>>> /usr/lib/locale/en_US.utf8/LC_COLLATE
>>> bash      20176        dem  mem       REG     252,0      286   1179929
>>> /usr/lib/locale/en_US.utf8/LC_MONETARY
>>> bash      20176        dem  mem       REG     252,0       52   1179930
>>> /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
>>> bash      20176        dem  mem       REG     252,0       34   1179931
>>> /usr/lib/locale/en_US.utf8/LC_PAPER
>>> bash      20176        dem  mem       REG     252,0       77   1179932
>>> /usr/lib/locale/en_US.utf8/LC_NAME
>>> bash      20176        dem  mem       REG     252,0      155   1179933
>>> /usr/lib/locale/en_US.utf8/LC_ADDRESS
>>> bash      20176        dem  mem       REG     252,0       59   1179934
>>> /usr/lib/locale/en_US.utf8/LC_TELEPHONE
>>> bash      20176        dem  mem       REG     252,0       23   1179935
>>> /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
>>> bash      20176        dem  mem       REG     252,0    26048    917676
>>> /usr/lib/gconv/gconv-modules.cache
>>> bash      20176        dem  mem       REG     252,0      373   1179936
>>> /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
>>>
>>> Version of Riak is 0.12.1. There was a similar problem once and the
>>> user was advised to make sure to use 0.12.1
>>>
>>> Any ideas?
>>>
>>> --
>>> Best regards,
>>> Dmitry Demeshchuk
>>>
>>
>>
>>
>> --
>> Best regards,
>> Dmitry Demeshchuk
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>



-- 
Best regards,
Dmitry Demeshchuk




More information about the riak-users mailing list