<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Ok - I think this turns out to be a GlusterFS 2.1.0-git specific bug,
but I've included all of the details:<br>
<br>
My first use of GlusterFS is using GlusterFS 2.1.0-git on Fedora 11 /
x86_64 with ext4 partitions. / is an ext4 partition.
/export/gluster-test is a different ext4 partition. I do use NFS +
AutoFS, and NFS does export other partitions under /export. This is to
be a very simple client/server.<br>
<br>
For the server, I have this:<br>
<br>
<tt># cat /export/gluster-test-server.vol <br>
volume brick<br>
type storage/posix <br>
option directory /export/gluster-test/<br>
end-volume<br>
<br>
volume server<br>
type protocol/server<br>
option transport-type tcp<br>
subvolumes brick<br>
option auth.addr.brick.allow 47.134.128.*<br>
end-volume<br>
</tt><br>
For the client, I have this:<br>
<br>
<tt># cat /export/gluster-test-client.vol<br>
volume brick1<br>
type protocol/client<br>
option transport-type tcp<br>
option remote-host 47.134.128.21<br>
#option remote-port 7000<br>
option remote-subvolume brick<br>
end-volume<br>
</tt><br>
The server/client IP is 47.134.128.21. There are no firewalls active on
this machine at this time.<br>
<br>
To launch the server, I used: (GlusterFS 2.1.0-git install int
/opt/glusterfs)<br>
<br>
<tt># /opt/glusterfs/sbin/glusterfsd
--volfile=/export/gluster-test-server.vol</tt><br>
<br>
To launch the client and mount, I used:<br>
<br>
<tt># mkdir /tmp/t<br>
# mount -t glusterfs /export/gluster-test-client.vol /tmp/t<br>
... output says FUSE initialized ...<br>
# cd /tmp/t</tt><br>
<br>
>From this point, I *appeared* to be able to modify /tmp/t. However, it
turns out that the mount did not actually complete, and I was just
changing /tmp/t under /tmp, not under GlusterFS. I believe this matches
the documented usage under gluster.org:<br>
<br>
<pre>bash# mount -t glusterfs /etc/glusterfs/glusterfs.vol /<i>mnt</i>/glusterfs
</pre>
<br>
I determined that I was able to sudo / su / login / run commands from
"/bin", however when I did "ls /" or "ls /export", everything would
freeze and "/sbin/shutdown -r now" would not complete. "cd /export"
would also freeze. I suspect that "ls /" does stat("/export") and this
is why it freezes. During this investigation period, I noticed the ps
output was strange:<br>
<br>
<tt>root 2312 1 0 16:10 ? 00:00:00
/opt/glusterfs/sbin/glusterfsd --volfile=gluster-test-server.vol<br>
root 2370 1 0 16:11 ? 00:00:00
/opt/glusterfs/sbin/glusterfs --log-level=NORMAL
--volfile=/export/gluster-test-client.vol /export<br>
root 2385 2370 0 16:11 ? 00:00:00 /bin/mount -i -f -t
fuse.glusterfs -o allow_other,default_permissions,max_read=131072
/export/gluster-test-client.vol /export<br>
root 2577 2467 0 16:13 tty4 00:00:00 grep gluster</tt><br>
<br>
Why is it trying to mount on /export?<br>
<br>
I ran this test multiple times - each time my ''mount -t glusterfs" was
on /tmp/t - I never used /export. Each time, it had the same results -
the /sbin/mount.glusterfs was somehow translating it to /export. I
determined to trace some of the processes, and found that the I could
"strace -p" for glusterfsd and glusterfs, but I could "strace -p" of
2385 would freeze. Control-C was frozen for all of these, including the
"strace -p", however, if I did "kill -9" (regular kill did not work) of
the /bin/mount process, then the "strace -p" would come back. Finally,
I killed /bin/mount *three* times (it came back twice?), and killed
glusterfs, the system went back to normal with no freezes. During this,
I also did a df on /tmp/t which showed that /tmp/t was /, but df in
general (which presumedly was trying to query /export) would freeze.<br>
<br>
To confirm this thinking, I started the glusterfs mount directly:<br>
<br>
# /opt/glusterfs/bin/glusterfs
--volfile=/export/gluster-test-client.vol /tmp/t<br>
<br>
And it worked perfectly - no freeze, and /tmp/t was a proper glusterfs
mount. Changes to /tmp/t were reflected in /export/gluster-test.<br>
<br>
I also determined that the complete system freeze and failure to
"/sbin/shutdown -r now" was due to failure for NFS to shut down
properly while the system was in the "frozen" state. If I restarted the
whole scenario, but ensured that both "nfs" and "autofs" were NOT
running, then although accesses to /export would freeze, I was able to
restart the system using "/sbin/shutdown -r now" or Ctrl-Alt-Del from
the console. So, the real freeze was that any access to /export would
become stuck in the kernel like an NFS hard mount. I did not wait
around to see if it would time out after 30 minutes as I was running
these tests in quick succession and my family was waiting for me
outside in the car. :-)<br>
<br>
Thinking about the above - I think /sbin/mount.glusterfs must have a
problem for it to use /export even though I passed in /tmp - but, this
is not the only problem. There is also some sort of other failure that
causes system lockup instead of clean failure. One scenario I can think
of is that it is trying to mount /export against something
/export/gluster-test, and this might be leading to some sort of loop? I
think /export was being put in a half-mounted state, where it was being
controlled by FUSE/GlusterFS, but GlusterFS was not able to serve any
requests on it?<br>
<br>
Going back to /sbin/mount.glusterfs, here is a more exact test showing
this problem:<br>
<br>
<tt>[root@wcarh033]/# mount -t glusterfs
/export/gluster-test-client.vol /tmp<br>
[root@wcarh033]/# ps -ef | grep gluster<br>
root 3221 1 0 17:54 ? 00:00:00
/opt/glusterfs/sbin/glusterfs --log-level=NORMAL
--volfile=/export/gluster-test-client.vol /export<br>
root 3232 3221 0 17:54 ? 00:00:00 /bin/mount -i -f -t
fuse.glusterfs -o allow_other,default_permissions,max_read=131072
/export/gluster-test-client.vol /export<br>
root 3238 3151 0 17:54 pts/0 00:00:00 grep gluster</tt><br>
<br>
If I try to recover from this, I can recover from the freeze, but not
from the whole situation:<br>
<br>
<tt>[root@wcarh033]/# kill -9 3221<br>
[root@wcarh033]/# ps -ef | grep gluster<br>
root 3232 1 0 17:54 ? 00:00:00 /bin/mount -i -f -t
fuse.glusterfs -o allow_other,default_permissions,max_read=131072
/export/gluster-test-client.vol /export<br>
root 3243 3151 0 17:56 pts/0 00:00:00 grep gluster<br>
[root@wcarh033]/# kill -9 3232<br>
[root@wcarh033]/# ps -ef | grep gluster<br>
root 3245 3151 0 17:56 pts/0 00:00:00 grep gluster<br>
[root@wcarh033]/# ls /export<br>
ls: cannot access /export: Transport endpoint is not connected</tt><br>
<br>
I reboot the machine to clean up for that, at least for now.<br>
<br>
Where is /export coming from? It's on the command line - I wonder if
the command line parsing is broken?<br>
<br>
In /sbin/mount.glusterfs, I see these lines which do not appear in
GlusterFS 2.0.6:<br>
<br>
<tt> mount_provided=$(echo "$@" | cut -f2 -d'/');<br>
<br>
[ -n "$mount_provided" ] && {<br>
mount_point="/$mount_provided";<br>
}<br>
<br>
[ -z "$mount_point" ] && {<br>
usage;<br>
exit 0;<br>
}<br>
<br>
</tt><br>
Before, it used to say:<br>
<br>
<tt> mount_point="$2";</tt><br>
<br>
If I switch the code back to what it used to be, my original test works
fine. No freeze. Whoohoo!<br>
<br>
Please fix in GIT. Thanks.<br>
<br>
Cheers,<br>
mark<br>
<br>
<pre class="moz-signature" cols="72">--
Mark Mielke <a class="moz-txt-link-rfc2396E" href="mailto:mark@mielke.cc"><mark@mielke.cc></a>
</pre>
</body>
</html>