It's a kernel bug which already fixed in RHEL6.2, try it with 2.6.32-220 kernel, it will works fine for you.<br><br><div class="gmail_quote">On Fri, May 11, 2012 at 12:24 PM, 程耀东 <span dir="ltr"><<a href="mailto:chyd@ihep.ac.cn" target="_blank">chyd@ihep.ac.cn</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I am using gluster in physical machine (CPU: 2 Xeon E5620, MEM: 24GB, 1Gpbs network link, Centos 6.0 linux 2.6.32-71.el6.x86_64). When reading or writing small numbers of files, the system is fine. But when too many files are accessing concurrently, the problem will occure some times. After disabling THP entirely:<br>
echo never> /sys/kernel/mm/redhat_transparent_hugepage/enabled<br>
It seems that the problem is resolved. I will continue to test it and see the results.<br>
<br>
Thank,<br>
yaodong<br>
<br>
> -----原始邮件-----<br>
> 发件人: "Bryan Whitehead" <<a href="mailto:driver@megahappy.net" target="_blank">driver@megahappy.net</a>><br>
> 发送时间: 2012年5月11日 星期五<br>
> 收件人: chyd <<a href="mailto:chyd@ihep.ac.cn" target="_blank">chyd@ihep.ac.cn</a>><br>
> 抄送: gluster-users <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
> 主题: Re: [Gluster-users] BUG: 764964 (dead lock)<br>
><br>
> Can you explain where glusterfs is being used? Is this lockup<br>
> happening on a VM running in on a file-disk-image on top of gluster?<br>
> is gluster itself causing this timeout?<br>
><br>
> On Wed, May 9, 2012 at 6:59 PM, chyd <<a href="mailto:chyd@ihep.ac.cn" target="_blank">chyd@ihep.ac.cn</a>> wrote:<br>
> > Hi all,<br>
> ><br>
> > I'm encountering a lockup problem many times when reading/writing large<br>
> > numbers of files. I cannot break out of the race in gdb, a ps will lock up<br>
> > when it tries to read that process' data, df (of course) locks up. No kill<br>
> > signals have any effect. Except 'pidstat -p ALL' can get the pid, I could't<br>
> > do anything. The only way out of it is to umount -f.<br>
> > I am using gluster 3.2.6 on CentOS 6.0 (2.6.32-71.el6.x86_64).<br>
> ><br>
> > The problem is the same as BUG 764964<br>
> > (<a href="https://bugzilla.redhat.com/show_bug.cgi?id=764964" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=764964</a>). and it is difficult to<br>
> > duplicate, I am find a way to produce it quickly. Any one else also<br>
> > encountered this problem? How do you solve it?<br>
> ><br>
> > Attached dmesg log:<br>
> > May 10 00:01:52 PPC-002 kernel: INFO: task glusterfs:27888 blocked for more<br>
> > than 120 seconds.<br>
> > May 10 00:01:52 PPC-002 kernel: "echo 0 ><br>
> > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
> > May 10 00:01:52 PPC-002 kernel: glusterfs D ffff88033fc24b00 0<br>
> > 27888 1 0x00000080<br>
> > May 10 00:01:52 PPC-002 kernel: ffff8806310fbe48 0000000000000086<br>
> > 0000000000000000 ffff8806310fbc58<br>
> > May 10 00:01:52 PPC-002 kernel: ffff8806310fbdc8 0000000000020010<br>
> > ffff8806310fbee8 00000001021f184a<br>
> > May 10 00:01:52 PPC-002 kernel: ffff8806311a0678 ffff8806310fbfd8<br>
> > 0000000000010518 ffff8806311a0678<br>
> > May 10 00:01:52 PPC-002 kernel: Call Trace:<br>
> > May 10 00:01:52 PPC-002 kernel: [<ffffffff814ca6b5>]<br>
> > rwsem_down_failed_common+0x95/0x1d0<br>
> > May 10 00:01:52 PPC-002 kernel: [<ffffffff814ca813>]<br>
> > rwsem_down_write_failed+0x23/0x30<br>
> > May 10 00:01:52 PPC-002 kernel: [<ffffffff81264253>]<br>
> > call_rwsem_down_write_failed+0x13/0x20<br>
> > May 10 00:01:52 PPC-002 kernel: [<ffffffff814c9d12>] ? down_write+0x32/0x40<br>
> > May 10 00:01:52 PPC-002 kernel: [<ffffffff8113b468>] sys_munmap+0x48/0x80<br>
> > May 10 00:01:52 PPC-002 kernel: [<ffffffff81013172>]<br>
> > system_call_fastpath+0x16/0x1b<br>
> ><br>
> > Thank you in advance.<br>
> > Yaodong<br>
> ><br>
> > 2012-05-10<br>
> > ________________________________<br>
> ><br>
> > _______________________________________________<br>
> > Gluster-users mailing list<br>
> > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> > <a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
> ><br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br>