<html><head></head><body bgcolor="#FFFFFF"><div><div>Hello Avati,</div><div><br></div><div>I got the log as below.</div><div>We can see that one gfid (g13) differs from others.</div><div>I sometimes found same situation so far.</div>
<div>Best regards</div><div>Kondo</div><div>------------------------</div><div><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">backend/ai/anchor/3:</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g10:trusted.gfid=0xd1366c6a09c345e8bab26b7953ac6405</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g11:trusted.gfid=0xd1366c6a09c345e8bab26b7953ac6405</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g12:trusted.gfid=0xd1366c6a09c345e8bab26b7953ac6405</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g13:trusted.gfid=0x93c1e7a2797a4b888e17d9b6582da365</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g14:trusted.gfid=0xd1366c6a09c345e8bab26b7953ac6405</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g15:trusted.gfid=0xd1366c6a09c345e8bab26b7953ac6405</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g16:trusted.gfid=0xd1366c6a09c345e8bab26b7953ac6405</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g17:trusted.gfid=0xd1366c6a09c345e8bab26b7953ac6405</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g18:trusted.gfid=0xd1366c6a09c345e8bab26b7953ac6405</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g10:trusted.glusterfs.dht=0x00000001000000003ffffffe5ffffffc</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g11:trusted.glusterfs.dht=0x00000001000000005ffffffd7ffffffb</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g12:trusted.glusterfs.dht=0x00000001000000007ffffffc9ffffffa</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g13:trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g14:trusted.glusterfs.dht=0x00000001000000009ffffffbbffffff9</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g15:trusted.glusterfs.dht=0x0000000100000000bffffffadffffff8</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g16:trusted.glusterfs.dht=0x0000000100000000dffffff9ffffffff</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g17:trusted.glusterfs.dht=0x0000000100000000000000001ffffffe</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g18:trusted.glusterfs.dht=0x00000001000000001fffffff3ffffffd</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US"> </span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">/backend/ai/anchor:</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g10:trusted.gfid=0x89a33935d5da4acd85586e075eca221d</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g11:trusted.gfid=0x89a33935d5da4acd85586e075eca221d</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g12:trusted.gfid=0x89a33935d5da4acd85586e075eca221d</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g13:trusted.gfid=0x6ee644c825dd4806b2680bfeb985a27d</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g14:trusted.gfid=0x89a33935d5da4acd85586e075eca221d</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g15:trusted.gfid=0x89a33935d5da4acd85586e075eca221d</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g16:trusted.gfid=0x89a33935d5da4acd85586e075eca221d</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g17:trusted.gfid=0x89a33935d5da4acd85586e075eca221d</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g18:trusted.gfid=0x89a33935d5da4acd85586e075eca221d</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g10:trusted.glusterfs.dht=0x00000001000000005555555471c71c6f</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g11:trusted.glusterfs.dht=0x000000010000000071c71c708e38e38b</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g12:trusted.glusterfs.dht=0x00000001000000008e38e38caaaaaaa7</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g13:trusted.glusterfs.dht=0x0000000100000000aaaaaaa8c71c71c3</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g14:trusted.glusterfs.dht=0x0000000100000000c71c71c4e38e38df</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g15:trusted.glusterfs.dht=0x0000000100000000e38e38e0ffffffff</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g16:trusted.glusterfs.dht=0x0000000100000000000000001c71c71b</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g17:trusted.glusterfs.dht=0x00000001000000001c71c71c38e38e37</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g18:trusted.glusterfs.dht=0x000000010000000038e38e3855555553</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US"> </span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">/backend/ai</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g10.gfid:trusted.gfid=0x258ba47b30ee41a984ceea1a491b1669</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g11.gfid:trusted.gfid=0x258ba47b30ee41a984ceea1a491b1669</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g12.gfid:trusted.gfid=0x258ba47b30ee41a984ceea1a491b1669</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g13.gfid:trusted.gfid=0x258ba47b30ee41a984ceea1a491b1669</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g14.gfid:trusted.gfid=0x258ba47b30ee41a984ceea1a491b1669</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g15.gfid:trusted.gfid=0x258ba47b30ee41a984ceea1a491b1669</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g16.gfid:trusted.gfid=0x258ba47b30ee41a984ceea1a491b1669</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g17.gfid:trusted.gfid=0x258ba47b30ee41a984ceea1a491b1669</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g18.gfid:trusted.gfid=0x258ba47b30ee41a984ceea1a491b1669</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g10.gfid:trusted.glusterfs.dht=0x00000001000000008e38e38caaaaaaa7</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g11.gfid:trusted.glusterfs.dht=0x0000000100000000aaaaaaa8c71c71c3</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g12.gfid:trusted.glusterfs.dht=0x0000000100000000c71c71c4e38e38df</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g13.gfid:trusted.glusterfs.dht=0x0000000100000000e38e38e0ffffffff</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g14.gfid:trusted.glusterfs.dht=0x0000000100000000000000001c71c71b</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g15.gfid:trusted.glusterfs.dht=0x00000001000000001c71c71c38e38e37</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US">g16.gfid:trusted.glusterfs.dht=0x000000010000000038e38e3855555553</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g17.gfid:trusted.glusterfs.dht=0x00000001000000005555555471c71c6f</span></p>
<p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt"><span lang="EN-US">g18.gfid:trusted.glusterfs.dht=0x000000010000000071c71c708e38e38b</span></p><p class="MsoNormal" style="text-align:-webkit-auto;margin-top:0mm;margin-right:0mm;margin-left:0mm;margin-bottom:0.0001pt">
<span lang="EN-US"> </span></p><span class="Apple-style-span" style="color:rgb(127,17,15);font-size:16px"><div><br></div><div><br></div></span></div><div><br>On 2012/12/04, at 9:20, Anand Avati &lt;<a href="mailto:anand.avati@gmail.com">anand.avati@gmail.com</a>&gt; wrote:<br>
<br></div><div></div><blockquote type="cite"><div><br><br><div class="gmail_quote">On Mon, Dec 3, 2012 at 5:14 AM, kenji kondo <span dir="ltr">&lt;<a href="mailto:kkay.jp@gmail.com" target="_blank">kkay.jp@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello Avati, thank you for your reply.<div><br></div><div>I tried to test your suggestion in 3.2.7, but I could not test in 3.3.x because I don&#39;t have.</div><div>For the results, unfortunately new similar problems occurred as following:</div>


<div><br></div><div><pre style="white-space:pre-wrap;word-wrap:break-word">gluster&gt; volume set vol22 performance.stat-prefetch off
gluster&gt; volume info vol22

Volume Name: vol22
Type: Distribute
Status: Started
Number of Bricks: 9
Transport-type: tcp
Bricks:
Brick1: gluster10:/export22/brick
Brick2: gluster11:/export22/brick
Brick3: gluster12:/export22/brick
Brick4: gluster13:/export22/brick
Brick5: gluster14:/export22/brick
Brick6: gluster15:/export22/brick
Brick7: gluster16:/export22/brick
Brick8: gluster17:/export22/brick
Brick9: gluster18:/export22/brick
Options Reconfigured:
performance.stat-prefetch: off

</pre></div><div>After this setting, I ran same simulation in the vol22.</div><div>But strange directories were made, that I could not remove the some directories as below:</div><div><br></div><div><pre style="white-space:pre-wrap;word-wrap:break-word">
$ ls -a ai/anchor/3
.  ..</pre><pre style="white-space:pre-wrap;word-wrap:break-word"><pre style="word-wrap:break-word;white-space:pre-wrap">$ rmdir ai/anchor/3
rmdir: ai/anchor/3: No such file or directory</pre><pre style="word-wrap:break-word;white-space:pre-wrap"><br></pre><pre style="word-wrap:break-word;white-space:pre-wrap">then I found error messages:</pre><pre style="word-wrap:break-word;white-space:pre-wrap">
[2012-12-03 18:08:14.816313] E [client3_1-fops.c:2228:client3_1_lookup_cbk]
0-vol22-client-3: remote operation failed: Stale NFS file handle
[2012-12-03 18:08:14.817196] W [dht-common.c:178:dht_lookup_dir_cbk]
0-vol22-dht: /test1130/ai/anchor: gfid different on vol22-client-4
[2012-12-03 18:08:14.817258] W [dht-common.c:178:dht_lookup_dir_cbk]
0-vol22-dht: /test1130/ai/anchor: gfid different on vol22-client-0
[2012-12-03 18:08:14.817322] W [dht-common.c:178:dht_lookup_dir_cbk]
0-vol22-dht: /test1130/ai/anchor: gfid different on vol22-client-2
[2012-12-03 18:08:14.817367] W [dht-common.c:178:dht_lookup_dir_cbk]
0-vol22-dht: /test1130/ai/anchor: gfid different on vol22-client-1
[2012-12-03 18:08:14.817398] W [dht-common.c:178:dht_lookup_dir_cbk]
0-vol22-dht: /test1130/ai/anchor: gfid different on vol22-client-5
[2012-12-03 18:08:14.817430] W [dht-common.c:178:dht_lookup_dir_cbk]
0-vol22-dht: /test1130/ai/anchor: gfid different on vol22-client-8
[2012-12-03 18:08:14.817460] W [dht-common.c:178:dht_lookup_dir_cbk]
0-vol22-dht: /test1130/ai/anchor: gfid different on vol22-client-7
[2012-12-03 18:08:14.817506] W [dht-common.c:178:dht_lookup_dir_cbk]
0-vol22-dht: /test1130/ai/anchor: gfid different on vol22-client-6
[2012-12-03 18:08:14.818865] E [client3_1-fops.c:2132:client3_1_opendir_cbk]
0-vol22-client-3: remote operation failed: No such file or directory
[2012-12-03 18:08:14.819198] W [fuse-bridge.c:1016:fuse_unlink_cbk]
0-glusterfs-fuse: 1684950: RMDIR()
/test1130/ai/anchor/3 =&gt; -1 (No such file or directory)
</pre><div><br></div><pre style="word-wrap:break-word;white-space:pre-wrap">And, I found strange dht with getfattr command,</pre><pre style="word-wrap:break-word;white-space:pre-wrap">$ sudo getfattr -d -m &#39;.*&#39; -n trusted.glusterfs.pathinfo ai/anchor/3<br>


</pre><pre style="word-wrap:break-word;white-space:pre-wrap">trusted.glusterfs.pathinfo=&quot;(vol22-dht-layout (vol22-client-0 1073741822
1610612732) (vol22-client-1 1610612733 2147483643) (vol22-client-2 2147483644
2684354554)</pre><pre style="word-wrap:break-word;white-space:pre-wrap"> (vol22-client-3 0 0) </pre><pre style="word-wrap:break-word;white-space:pre-wrap">(vol22-client-4 2684354555 3221225465)
(vol22-client-5 3221225466 3758096376) (vol22-client-6 3758096377 4294967295)
(vol22-client-7 0 536870910) (vol22-client-8 536870911 1073741821))&quot;
</pre><div><br></div></pre></div><div>vol22-client-3 is 0 to 0?: this would be incorrect.</div><div><br></div><div>Above problem is found in all of our clients.</div><div>I expect that the problems relate to exclusive control for mkdir, but I don&#39;t enough understand this phenomenon.<br>


</div><div><br></div><div>Can you have some idea?</div><div>I will try.</div><div><br></div></blockquote><div><br></div><div>In such a state, can you get the backend xattr dumps of ai/anchor/ and ai/anchor/3/ with the following commands run on ALL servers -</div>

<div><br></div><div>sh# getfattr -d -e hex -m . /backend/ai/anchor/</div><div>sh# getfattr -d -e hex -m . /backend/ai/anchor/3/</div><div><br></div><div>Thanks,</div><div>Avati</div><div><br></div></div>
</div></blockquote></div><div><span></span></div></body></html>