File descriptor leaks?

Justin Sheehy justin at basho.com
Mon Oct 18 10:11:09 EDT 2010


Hi, Dmitry.

What version of Riak are you using?  And is there anything interesting
in the error logs?

-Justin




On Thu, Oct 14, 2010 at 7:53 AM, Dmitry Demeshchuk <demeshchuk at gmail.com> wrote:
> A small update. I've just encountered the same problem. Just about 3-4
> hours have passed.
>
> lsof | wc -l showed only about 2k descriptors for all users. That's
> even more weird as the 32k descriptors limit is per user. So, we
> haven't reached the limit so far.
>
> On Thu, Oct 14, 2010 at 3:48 PM, Dmitry Demeshchuk <demeshchuk at gmail.com> wrote:
>> Greetings.
>>
>> We have recently started to get the emfile errors. ulimit -n is 32767.
>> Restarting Riak helps for several hours and then we run out of
>> descriptors again.
>>
>> Some time later after restart I performed lsof and found the following
>> descriptors:
>>
>> kondemand   154       root  cwd   unknown
>> /proc/154/cwd (readlink: Permission denied)
>> kondemand   154       root  rtd   unknown
>> /proc/154/root (readlink: Permission denied)
>> kondemand   154       root  txt   unknown
>> /proc/154/exe (readlink: Permission denied)
>> kondemand   154       root NOFD
>> /proc/154/fd (opendir: Permission denied)
>> kondemand   155       root  cwd   unknown
>> /proc/155/cwd (readlink: Permission denied)
>> kondemand   154       root  cwd   unknown
>> /proc/154/cwd (readlink: Permission denied)
>> kondemand   154       root  rtd   unknown
>> /proc/154/root (readlink: Permission denied)
>> kondemand   154       root  txt   unknown
>> /proc/154/exe (readlink: Permission denied)
>> kondemand   154       root NOFD
>> /proc/154/fd (opendir: Permission denied)
>> kondemand   155       root  cwd   unknown
>> /proc/155/cwd (readlink: Permission denied)
>> kondemand   155       root  rtd   unknown
>> /proc/155/root (readlink: Permission denied)
>> kondemand   155       root  txt   unknown
>> /proc/155/exe (readlink: Permission denied)
>> kondemand   155       root NOFD
>> /proc/155/fd (opendir: Permission denied)
>> kondemand   156       root  cwd   unknown
>> /proc/156/cwd (readlink: Permission denied)
>> kondemand   156       root  rtd   unknown
>> /proc/156/root (readlink: Permission denied)
>> kondemand   156       root  txt   unknown
>> /proc/156/exe (readlink: Permission denied)
>> kondemand   156       root NOFD
>> /proc/156/fd (opendir: Permission denied)
>> kondemand   157       root  cwd   unknown
>> /proc/157/cwd (readlink: Permission denied)
>> kondemand   157       root  rtd   unknown
>> /proc/157/root (readlink: Permission denied)
>> kondemand   157       root  txt   unknown
>> /proc/157/exe (readlink: Permission denied)
>> kondemand   157       root NOFD
>> /proc/157/fd (opendir: Permission denied)
>> kondemand   158       root  cwd   unknown
>> /proc/158/cwd (readlink: Permission denied)
>> kondemand   158       root  rtd   unknown
>> /proc/158/root (readlink: Permission denied)
>> kondemand   158       root  txt   unknown
>> /proc/158/exe (readlink: Permission denied)
>>
>> Also, the following couple of descriptors is opened several times at
>> the same time:
>>
>> bash      20176        dem  mem       REG     252,0   256316   1179925
>> /usr/lib/locale/en_US.utf8/LC_CTYPE
>> bash      20176        dem  mem       REG     252,0       54   1179926
>> /usr/lib/locale/en_US.utf8/LC_NUMERIC
>> bash      20176        dem  mem       REG     252,0     2454   1179927
>> /usr/lib/locale/en_US.utf8/LC_TIME
>> bash      20176        dem  mem       REG     252,0   966938   1179928
>> /usr/lib/locale/en_US.utf8/LC_COLLATE
>> bash      20176        dem  mem       REG     252,0      286   1179929
>> /usr/lib/locale/en_US.utf8/LC_MONETARY
>> bash      20176        dem  mem       REG     252,0       52   1179930
>> /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
>> bash      20176        dem  mem       REG     252,0       34   1179931
>> /usr/lib/locale/en_US.utf8/LC_PAPER
>> bash      20176        dem  mem       REG     252,0       77   1179932
>> /usr/lib/locale/en_US.utf8/LC_NAME
>> bash      20176        dem  mem       REG     252,0      155   1179933
>> /usr/lib/locale/en_US.utf8/LC_ADDRESS
>> bash      20176        dem  mem       REG     252,0       59   1179934
>> /usr/lib/locale/en_US.utf8/LC_TELEPHONE
>> bash      20176        dem  mem       REG     252,0       23   1179935
>> /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
>> bash      20176        dem  mem       REG     252,0    26048    917676
>> /usr/lib/gconv/gconv-modules.cache
>> bash      20176        dem  mem       REG     252,0      373   1179936
>> /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
>>
>> Version of Riak is 0.12.1. There was a similar problem once and the
>> user was advised to make sure to use 0.12.1
>>
>> Any ideas?
>>
>> --
>> Best regards,
>> Dmitry Demeshchuk
>>
>
>
>
> --
> Best regards,
> Dmitry Demeshchuk
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>




More information about the riak-users mailing list