<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><span style="white-space: pre-wrap; ">Hey gluster-users,</span></div><div><span style="white-space: pre-wrap; ">I just stumbled on a problem in our current test-setup of gluster 3.3.2.</span></div><div><span style="white-space: pre-wrap; "><br></span></div><div><span style="white-space: pre-wrap; ">This is a simple replicated setup with 2 bricks (on XFS) in 1 volume running on </span>glusterfs version 3.3.2qa3 on ubuntu lucid<span style="white-space: pre-wrap; ">.</span></div><div><span style="white-space: pre-wrap; ">The client mounting this volume on /mnt/gfs sits on a mother machine and is using fuse (</span>Version: 2.8.1-1.1ubuntu3.1)<span style="white-space: pre-wrap; ">.</span></div><div><span style="white-space: pre-wrap;"><br></span></div><div><span style="white-space: pre-wrap;">On the gluster-fs fuse client mount log:</span></div><div><span style="white-space: pre-wrap;">[2013-06-02 21:23:26.677069] W [afr-common.c:1196:afr_detect_self_heal_by_iatt] 0-test-fs-cluster-1-replicate-0: /home/filesshared/README.txt.lock: gfid different on subvolume</span></div><div><span style="white-space: pre-wrap;">[2013-06-02 21:23:26.677069] I [afr-self-heal-common.c:1970:afr_sh_post_nb_entrylk_gfid_sh_cbk] 0-test-fs-cluster-1-replicate-0: Non blocking entrylks failed.</span></div><div><span style="white-space: pre-wrap;">[2013-06-02 21:23:26.697068] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-test-fs-cluster-1-client-0: remote operation failed: File exists. Path: /home/filesshared/README.txt.lock (00000000-0000-0000-0000-000000000000)</span></div><div><span style="white-space: pre-wrap;">[2013-06-02 21:23:26.697068] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-test-fs-cluster-1-client-1: remote operation failed: File exists. Path: /home/filesshared/README.txt.lock (00000000-0000-0000-0000-000000000000)</span></div><div><span style="white-space: pre-wrap;">[2013-06-02 21:23:26.697068] W [inode.c:914:inode_lookup] (-->/usr/lib/glusterfs/3.3.2qa3/xlator/debug/io-stats.so(io_stats_lookup_cbk+0xff) [0x7fb16c310d8f] (-->/usr/lib/glusterfs/3.3.2qa3/xlator/mount/fuse.so(+0xf248) [0x7fb16fa95248] (-->/usr/lib/glusterfs/3.3.2qa3/xlator/mount/fuse.so(+0xf0b1) [0x7fb16fa950b1]))) 0-fuse: inode not found</span></div><div><span style="white-space: pre-wrap;"><br></span></div><div><br></div><div><span style="white-space: pre-wrap;">What the application side is doing when this happened:</span></div><div><span style="white-space: pre-wrap;">1. It created /home/filesshared</span></div><div><span style="white-space: pre-wrap;">2. creates /mnt/gfs/home/filesshared</span></div><div><span style="white-space: pre-wrap;">3. deleted /home/filesshared and replaced it with a symlink from /home/filesshared to /mnt/gfs/home/filesshared</span></div><div><span style="white-space: pre-wrap;">4. Tried to write some files</span></div><div><br></div><div>Here's the log for that:</div><div><span style="white-space: pre-wrap;">2013-06-02T21:23:26+00:00 daemon.notice web-14 f-c-w[4842]: deploying filesshared.prod</span></div><div><span style="white-space: pre-wrap;">2013-06-02T21:23:26+00:00 daemon.notice web-14 f-c-w[4842]: creating directory: dir=/home/filesshared, user=0, group=filesshared, mode=0550</span></div><div><span style="white-space: pre-wrap;">2013-06-02T21:23:26+00:00 daemon.notice web-14 f-c-w[4842]: creating directory: dir=/mnt/gfs/home/filesshared, user=filesshared, group=filesshared, mode=0700</span></div><div><span style="white-space: pre-wrap;">2013-06-02T21:23:26+00:00 daemon.notice web-14 f-c-w[4842]: created /home/filesshared -> /mnt/gfs/home/filesshared</span></div><div><span style="white-space: pre-wrap;">2013-06-02T21:23:26+00:00 daemon.notice web-14 f-c-w[4842]: PHP Warning: stat(): stat failed for /home/filesshared/README.txt.lock in /usr/ah/lib/ah-lib.php on line 701</span></div><div><span style="white-space: pre-wrap;">2013-06-02T21:23:27+00:00 daemon.notice web-14 f-c-w[4842]: PHP Warning: stat(): stat failed for /home/filesshared/README.txt.lock in /usr/ah/lib/ah-lib.php on line 701</span></div><div><span style="white-space: pre-wrap;">2013-06-02T21:23:27+00:00 daemon.notice web-14 f-c-w[4842]: PHP Warning: stat(): stat failed for /home/filesshared/README.txt.lock in /usr/ah/lib/ah-lib.php on line 701</span></div><div><span style="white-space: pre-wrap;">2013-06-02T21:23:28+00:00 daemon.notice web-14 f-c-w[4842]: PHP Warning: stat(): stat failed for /home/filesshared/README.txt.lock in /usr/ah/lib/ah-lib.php on line 701</span></div><div><span style="white-space: pre-wrap;"><br></span></div><div><span style="white-space: pre-wrap;">What this resulted in:</span></div><div><span style="white-space: pre-wrap;">This turned the mount point completely unresponsive.</span></div><div><span style="white-space: pre-wrap;">This means that in PHP, file_exists('/mnt/gfs') returns false and stat() calls fail. </span><span style="white-space: pre-wrap; ">In Ruby File.directory?('/mnt/gfs') returns false.</span></div><div><span style="white-space: pre-wrap;">This can be solved by calling "umount /mnt/gfs" and then remounting the share again from fstab ("mount /mnt/gfs")</span></div><div><span style="white-space: pre-wrap;"><br></span></div><div><span style="white-space: pre-wrap;">I could not find any relevant log entries on the bricks themselves. I sadly also wasn't able to come up with a test case to reproduce it.</span></div><div><br></div><div><span style="white-space: pre-wrap;">It seems somewhat similar to </span><a href="http://gluster.org/pipermail/gluster-users/2013-March/035662.html">http://gluster.org/pipermail/gluster-users/2013-March/035662.html</a></div><div>I initially thought that this could have been fixed in <a href="http://review.gluster.org/#/c/4689/">http://review.gluster.org/#/c/4689/</a> , but the qa branch we run has this fix backported.</div><div><span style="white-space: pre-wrap;"><br></span></div><div><span style="white-space: pre-wrap; ">Any idea what could cause this </span><span style="white-space: pre-wrap; ">behaviour</span><span style="white-space: pre-wrap; ">?</span></div><div><span style="white-space: pre-wrap;"><br></span></div><div><span style="white-space: pre-wrap;">Cheers,</span></div><div><span style="white-space: pre-wrap;">Marc</span></div></body></html>