<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jul 18, 2014 at 10:43 PM, Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class=""><br>
On 07/18/2014 07:57 PM, Anders Blomdell wrote:<br>
</div><div class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
During testing of a 3*4 gluster (from master as of yesterday), I encountered<br>
two major weirdnesses:<br>
<br>
1. A 'rm -rf <some_dir>' needed several invocations to finish, each time<br>
reporting a number of lines like these:<br>
rm: cannot remove ‘a/b/c/d/e/f’: Directory not empty<br></blockquote></div></blockquote><div><br></div><div>This is reproducible for me when running dbench on nfs mounts. I think I may have seen it on glusterfs mounts as well but it seems more reproducible on nfs. I should have caught it sooner but it doesn't error out client side when cleaning up, and the next test I run the deletes are successful. When this happens in the nfs.log I see:</div>
<div><br></div><div>This spams the log, from what I can tell it happens when dbench is creating the files:</div><div>[2014-07-19 13:37:03.271651] I [MSGID: 109036] [dht-common.c:5694:dht_log_new_layout_for_dir_selfheal] 0-testvol-dht: Setting layout of /clients/client3/~dmtmp/SEED with [Subvol_name: testvol-replicate-0, Err: -1 , Start: 2147483647 , Stop: 4294967295 ], [Subvol_name: testvol-replicate-1, Err: -1 , Start: 0 , Stop: 2147483646 ],<br>
</div><div><br></div><div>Then when the deletes fail I see the following when the client is removing the files:</div><div>[2014-07-18 23:31:44.272465] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 74a6541a: /run8063_dbench/clients => -1 (Directory not empty)<br>
</div><div><div>.</div><div>.</div><div>[2014-07-18 23:31:44.452988] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 7ea9541a: /run8063_dbench/clients => -1 (Directory not empty)</div><div>[2014-07-18 23:31:45.262651] W [client-rpc-fops.c:1354:client3_3_access_cbk] 0-testvol-client-0: remote operation failed: Stale file handle</div>
<div>[2014-07-18 23:31:45.263151] W [MSGID: 108008] [afr-read-txn.c:218:afr_read_txn] 0-testvol-replicate-0: Unreadable subvolume -1 found with e</div><div>vent generation 2. (Possible split-brain)</div><div>[2014-07-18 23:31:45.264196] W [nfs3.c:1532:nfs3svc_access_cbk] 0-nfs: 32ac541a: <gfid:b073a189-91ea-46b2-b757-5b320591b848> => -1 (Stale fi</div>
<div>le handle)</div><div>[2014-07-18 23:31:45.264217] W [nfs3-helpers.c:3401:nfs3_log_common_res] 0-nfs-nfsv3: XID: 32ac541a, ACCESS: NFS: 70(Invalid file handle), P</div><div>OSIX: 116(Stale file handle)</div><div>[2014-07-18 23:31:45.266818] W [nfs3.c:1532:nfs3svc_access_cbk] 0-nfs: 33ac541a: <gfid:b073a189-91ea-46b2-b757-5b320591b848> => -1 (Stale fi</div>
<div>le handle)</div><div>[2014-07-18 23:31:45.266853] W [nfs3-helpers.c:3401:nfs3_log_common_res] 0-nfs-nfsv3: XID: 33ac541a, ACCESS: NFS: 70(Invalid file handle), P</div><div>OSIX: 116(Stale file handle)</div></div><div>
<br></div><div>Occasionally I see:</div><div><div>[2014-07-19 13:50:46.091429] W [socket.c:529:__socket_rwv] 0-NLM-client: readv on <a href="http://192.168.11.102:45823">192.168.11.102:45823</a> failed (No data available)</div>
<div>[2014-07-19 13:50:46.091570] E [rpc-transport.c:485:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/nfs/server.so(nlm_rpcclnt_notify+0x5a) [0x7f53775128ea] (-->/usr/lib64/glusterfs/3.5qa2/xlator/nfs/server.so(nlm_unset_rpc_clnt+0x75) [0x7f537750e3e5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x7f5388914693]))) 0-rpc_transport: invalid argument: this</div>
</div><div><br></div><div>I'm opening a BZ now, I'll leave systems up and put the repro steps + hostnames in the BZ in case anyone wants to poke around.</div><div><br></div><div>-b</div><div><br></div><div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
2. After having successfully deleted all files from the volume,<br>
i have a single directory that is duplicated in gluster-fuse,<br>
like this:<br>
# ls -l /mnt/gluster<br>
total 24<br>
drwxr-xr-x 2 root root 12288 18 jul 16.17 work2/<br>
drwxr-xr-x 2 root root 12288 18 jul 16.17 work2/<br>
<br>
any idea on how to debug this issue?<br>
</blockquote></div>
What are the steps to recreate? We need to first find what lead to this. Then probably which xlator leads to this.<br></blockquote><div><br></div><div>I have not seen this but I am running on a 6x2 volume. I wonder if this may only happen with replica > 2?</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Pranith<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
/Anders<br>
<br>
</blockquote><div class=""><div class="h5">
<br>
______________________________<u></u>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.<u></u>org/mailman/listinfo/gluster-<u></u>devel</a><br>
</div></div></blockquote></div><br></div></div>