<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:宋体;}
pre
{mso-style-priority:99;
mso-style-link:"HTML 预设格式 Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"批注框文本 Char";
margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:9.0pt;
font-family:"Calibri","sans-serif";}
span.Char
{mso-style-name:"批注框文本 Char";
mso-style-priority:99;
mso-style-link:批注框文本;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.HTMLChar
{mso-style-name:"HTML 预设格式 Char";
mso-style-priority:99;
mso-style-link:"HTML 预设格式";
font-family:"Courier New";}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=ZH-CN link=blue vlink=purple style='text-justify-trim:punctuation'><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>Joe,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>I test it again, dump related glusterfs info and create a bug report on bugzilla.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> </span><span lang=EN-US style='color:#1F497D'><a href="https://bugzilla.redhat.com/show_bug.cgi?id=906238">https://bugzilla.redhat.com/show_bug.cgi?id=906238</a><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>I use "kill -USR1 <hanged glusterfs client process ID>" to dump info and find that "gfs28-replicate-5" maybe be hanged. Then, I dump glusterfsd info of "Brick16: 10.1.10.188:/xmail/disk2/gfs28" and find the "/xmail/disk2/gfs28/songcl/b83/003.txt" is opened two times by "ls -asl /proce/pid/fd" command. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>Maybe this file is deadlocked according to corresponding glusterfsd log:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>[2013-01-31 13:42:20.927077] T [rpcsvc.c:187:rpcsvc_program_actor] 0-rpc-service: Actor found: GlusterFS 3.2.7 - INODELK<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>[2013-01-31 13:42:20.927090] T [server-resolve.c:127:resolve_loc_touchup] 0-gfs28-server: return value inode_path 11<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>[2013-01-31 13:42:20.927104] T [common.c:103:get_domain] 0-posix-locks: Domain gfs28-replicate-5 found<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>[2013-01-31 13:42:20.927113] T [inodelk.c:218:__lock_inodelk] 0-gfs28-locks: Lock (pid=1059928640) lk-owner:140197382404672 9223372036854775806 - 0 => Blocked<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>[2013-01-31 13:42:20.927123] T [inodelk.c:486:pl_inode_setlk] 0-gfs28-locks: Lock (pid=1059928640) (lk-owner=140197382404672) 9223372036854775806 - 0 => NOK<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>[2013-01-31 13:42:20.927132] T [inodelk.c:218:__lock_inodelk] 0-gfs28-locks: Lock (pid=1059928640) lk-owner:140197382404672 9223372036854775806 - 0 => Blocked<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>[2013-01-31 13:42:20.933429] T [rpcsvc.c:443:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 987<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>For more information, please refer to attachment.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>1. PID:6988 is the hanged glusterfs client dump file.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>2. PID:31100 is the glusterfsd dump file of "Brick16".<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>3. 188-xmail-disk2-gfs28.log.splitab is the glusterfsd log of "Brick16".<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>If you need any other debug information, please tell me. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'>Thanks very much!<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal align=left style='text-align:left'><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Joe Julian [mailto:joe@julianfamily.org] <br><b>Sent:</b> Friday, January 25, 2013 12:15 AM<br><b>To:</b> Song; gluster-devel@nongnu.org<br><b>Subject:</b> Re: [Gluster-devel] glusterfs(3.2.7) hang when making the same dir at the same time<o:p></o:p></span></p></div></div><p class=MsoNormal align=left style='text-align:left'><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal align=left style='margin-bottom:12.0pt;text-align:left'><span lang=EN-US style='font-size:12.0pt;font-family:宋体'>This looks like a support question to me. If you are asking a development question, you might want to use strace or gdb to figure out where the hang is, file a bug report on bugzilla, and submit your patch(es) to gerrit. <o:p></o:p></span></p><div><p class=MsoNormal align=left style='text-align:left'><span lang=EN-US style='font-size:12.0pt;font-family:宋体'>Song <<a href="mailto:gluster@163.com">gluster@163.com</a>> wrote:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Hi,<o:p></o:p></span></p><p><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Recently, glusterfs will hang when we do stress testing. To find the reason, we write a test shell script.<o:p></o:p></span></p><p><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>We run the test shell on 5 servers at the same time. For a moment, all test programming is hang.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>When execute command “cd /xmail/gfs1/scl_test/001”, also hang.<o:p></o:p></span></p><p><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>The test shell script:<o:p></o:p></span></p><p><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal><i><span lang=EN-US>for((i=1;i<=100;i++));<o:p></o:p></span></i></p><p class=MsoNormal><i><span lang=EN-US>do <o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>rmdir /xmail/gfs1/scl_test/001<o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>if [ "$?" == "0" ];<o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>then <o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:21.0pt'><i><span lang=EN-US>echo "delete dir success"<o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>fi <o:p></o:p></span></i></p><p><i><span lang=EN-US> <o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>mkdir /xmail/gfs1/scl_test/001<o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>if [ "$?" == "0" ];<o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>then <o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:21.0pt'><i><span lang=EN-US>echo "create dir success"<o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>fi<o:p></o:p></span></i></p><p><i><span lang=EN-US> <o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>echo "1111" >>/xmail/gfs1/scl_test/001/001.txt<o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>echo "2222" >>/xmail/gfs1/scl_test/001/002.txt<o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>echo "3333" >>/xmail/gfs1/scl_test/001/003.txt<o:p></o:p></span></i></p><p><i><span lang=EN-US> <o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>rm -rf /xmail/gfs1/scl_test/001/001.txt<o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>rm -ff /xmail/gfs1/scl_test/001/002.txt<o:p></o:p></span></i></p><p class=MsoNormal style='text-indent:10.5pt'><i><span lang=EN-US>rm -rf /xmail/gfs1/scl_test/001/003.txt<o:p></o:p></span></i></p><p class=MsoNormal><i><span lang=EN-US>done<o:p></o:p></span></i></p><p><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>“/xmail/gfs1” is native mount point of gluster volume gfs1.<o:p></o:p></span></p><p><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Gluster volume info is as below:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[root@d181 glusterfs]# gluster volume info<o:p></o:p></span></p><p><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Volume Name: gfs1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Type: Distributed-Replicate<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Status: Started<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Number of Bricks: 30 x 3 = 90<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Transport-type: tcp<o:p></o:p></span></p><p><span lang=EN-US> <o:p></o:p></span></p><p><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Please help me, Thanks!<o:p></o:p></span></p><p><span lang=EN-US> <o:p></o:p></span></p><pre style='text-align:center;white-space:pre-wrap;word-wrap:break-word'><span lang=EN-US style='font-family:"Arial","sans-serif"'><hr size=2 width="100%" align=center></span></pre><pre><span lang=EN-US style='font-family:"Arial","sans-serif"'><br>Gluster-devel mailing list<br><a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br><a href="https://lists.nongnu.org/mailman/listinfo/gluster-devel">https://lists.nongnu.org/mailman/listinfo/gluster-devel</a><o:p></o:p></span></pre></div></div></body></html>