Can you also give the outputs of "getfattr -d -m . -e hex /backend/dir" from each of the bricks? It will be interesting to know in case there was a gfid mismatch somehow.<div><br></div><div>Avati<br><br><div class="gmail_quote">
On Thu, Jan 31, 2013 at 1:47 AM, Song <span dir="ltr"><<a href="mailto:gluster@163.com" target="_blank">gluster@163.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="ZH-CN" link="blue" vlink="purple"><div><p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">Joe,<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">I test it again, dump related glusterfs info and create a bug report on bugzilla.<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US"> </span><span lang="EN-US" style="color:#1f497d"><a href="https://bugzilla.redhat.com/show_bug.cgi?id=906238" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=906238</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">I use "kill -USR1 <hanged glusterfs client process ID>" to dump info and find that "gfs28-replicate-5" maybe be hanged. Then, I dump glusterfsd info of "Brick16: 10.1.10.188:/xmail/disk2/gfs28" and find the "/xmail/disk2/gfs28/songcl/b83/003.txt" is opened two times by "ls -asl /proce/pid/fd" command. <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">Maybe this file is deadlocked according to corresponding glusterfsd log:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">[2013-01-31 13:42:20.927077] T [rpcsvc.c:187:rpcsvc_program_actor] 0-rpc-service: Actor found: GlusterFS 3.2.7 - INODELK<u></u><u></u></span></p><p class="MsoNormal">
<span lang="EN-US" style="color:#1f497d">[2013-01-31 13:42:20.927090] T [server-resolve.c:127:resolve_loc_touchup] 0-gfs28-server: return value inode_path 11<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">[2013-01-31 13:42:20.927104] T [common.c:103:get_domain] 0-posix-locks: Domain gfs28-replicate-5 found<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">[2013-01-31 13:42:20.927113] T [inodelk.c:218:__lock_inodelk] 0-gfs28-locks: Lock (pid=1059928640) lk-owner:140197382404672 9223372036854775806 - 0 => Blocked<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">[2013-01-31 13:42:20.927123] T [inodelk.c:486:pl_inode_setlk] 0-gfs28-locks: Lock (pid=1059928640) (lk-owner=140197382404672) 9223372036854775806 - 0 => NOK<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">[2013-01-31 13:42:20.927132] T [inodelk.c:218:__lock_inodelk] 0-gfs28-locks: Lock (pid=1059928640) lk-owner:140197382404672 9223372036854775806 - 0 => Blocked<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">[2013-01-31 13:42:20.933429] T [rpcsvc.c:443:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 987<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">For more information, please refer to attachment.<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">1. PID:6988 is the hanged glusterfs client dump file.<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">2. PID:31100 is the glusterfsd dump file of "Brick16".<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">3. 188-xmail-disk2-gfs28.log.splitab is the glusterfsd log of "Brick16".<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">If you need any other debug information, please tell me. <u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">Thanks very much!<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d"><u></u> <u></u></span></p><div><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal" align="left" style="text-align:left">
<b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Joe Julian [mailto:<a href="mailto:joe@julianfamily.org" target="_blank">joe@julianfamily.org</a>] <br>
<b>Sent:</b> Friday, January 25, 2013 12:15 AM<br><b>To:</b> Song; <a href="mailto:gluster-devel@nongnu.org" target="_blank">gluster-devel@nongnu.org</a><br><b>Subject:</b> Re: [Gluster-devel] glusterfs(3.2.7) hang when making the same dir at the same time<u></u><u></u></span></p>
</div></div><div><div class="h5"><p class="MsoNormal" align="left" style="text-align:left"><span lang="EN-US"><u></u> <u></u></span></p><p class="MsoNormal" align="left" style="margin-bottom:12.0pt;text-align:left"><span lang="EN-US" style="font-size:12.0pt;font-family:宋体">This looks like a support question to me. If you are asking a development question, you might want to use strace or gdb to figure out where the hang is, file a bug report on bugzilla, and submit your patch(es) to gerrit. <u></u><u></u></span></p>
<div><p class="MsoNormal" align="left" style="text-align:left"><span lang="EN-US" style="font-size:12.0pt;font-family:宋体">Song <<a href="mailto:gluster@163.com" target="_blank">gluster@163.com</a>> wrote:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Hi,<u></u><u></u></span></p><p><span lang="EN-US"> <u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US">Recently, glusterfs will hang when we do stress testing. To find the reason, we write a test shell script.<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US">We run the test shell on 5 servers at the same time. For a moment, all test programming is hang.<u></u><u></u></span></p><p class="MsoNormal">
<span lang="EN-US">When execute command “cd /xmail/gfs1/scl_test/001”, also hang.<u></u><u></u></span></p><p><span lang="EN-US"> <u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US">The test shell script:<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p><p class="MsoNormal"><i><span lang="EN-US">for((i=1;i<=100;i++));<u></u><u></u></span></i></p><p class="MsoNormal"><i><span lang="EN-US">do <u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:10.5pt">
<i><span lang="EN-US">rmdir /xmail/gfs1/scl_test/001<u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:10.5pt"><i><span lang="EN-US">if [ "$?" == "0" ];<u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:10.5pt">
<i><span lang="EN-US">then <u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:21.0pt"><i><span lang="EN-US">echo "delete dir success"<u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:10.5pt">
<i><span lang="EN-US">fi <u></u><u></u></span></i></p><p><i><span lang="EN-US"> <u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:10.5pt"><i><span lang="EN-US">mkdir /xmail/gfs1/scl_test/001<u></u><u></u></span></i></p>
<p class="MsoNormal" style="text-indent:10.5pt"><i><span lang="EN-US">if [ "$?" == "0" ];<u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:10.5pt"><i><span lang="EN-US">then <u></u><u></u></span></i></p>
<p class="MsoNormal" style="text-indent:21.0pt"><i><span lang="EN-US">echo "create dir success"<u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:10.5pt"><i><span lang="EN-US">fi<u></u><u></u></span></i></p>
<p><i><span lang="EN-US"> <u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:10.5pt"><i><span lang="EN-US">echo "1111" >>/xmail/gfs1/scl_test/001/001.txt<u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:10.5pt">
<i><span lang="EN-US">echo "2222" >>/xmail/gfs1/scl_test/001/002.txt<u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:10.5pt"><i><span lang="EN-US">echo "3333" >>/xmail/gfs1/scl_test/001/003.txt<u></u><u></u></span></i></p>
<p><i><span lang="EN-US"> <u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:10.5pt"><i><span lang="EN-US">rm -rf /xmail/gfs1/scl_test/001/001.txt<u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:10.5pt">
<i><span lang="EN-US">rm -ff /xmail/gfs1/scl_test/001/002.txt<u></u><u></u></span></i></p><p class="MsoNormal" style="text-indent:10.5pt"><i><span lang="EN-US">rm -rf /xmail/gfs1/scl_test/001/003.txt<u></u><u></u></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US">done<u></u><u></u></span></i></p><p><span lang="EN-US"> <u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US">“/xmail/gfs1” is native mount point of gluster volume gfs1.<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US">Gluster volume info is as below:<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US">[root@d181 glusterfs]# gluster volume info<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US">Volume Name: gfs1<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US">Type: Distributed-Replicate<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Status: Started<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US">Number of Bricks: 30 x 3 = 90<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US">Transport-type: tcp<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p><p><span lang="EN-US"> <u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US">Please help me, Thanks!<u></u><u></u></span></p><p><span lang="EN-US"> <u></u><u></u></span></p>
<pre style="text-align:center;white-space:pre-wrap;word-wrap:break-word"><span lang="EN-US" style="font-family:"Arial","sans-serif""><hr size="2" width="100%" align="center"></span></pre><pre><span lang="EN-US" style="font-family:"Arial","sans-serif""><br>
Gluster-devel mailing list<br><a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br><a href="https://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">https://lists.nongnu.org/mailman/listinfo/gluster-devel</a><u></u><u></u></span></pre>
</div></div></div></div></div><br>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
<a href="https://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">https://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
<br></blockquote></div><br></div>