Fred,<br> Can you also provide us server logs?<br> <br>--<br>gowda<br><br><br><div class="gmail_quote">On Tue, Nov 25, 2008 at 4:57 PM, Fred Hucht <span dir="ltr"><<a href="mailto:fred@thp.uni-due.de">fred@thp.uni-due.de</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi devels!<br>
<br>
We consider GlusterFS as parallel file server (8 server nodes) for our parallel Opteron cluster (88 nodes, ~500 cores), as well as for a unified nufa /scratch distributed over all nodes. We use the cluster within a scientific environment (theoretical physics) and use Scientific Linux with kernel <a href="http://2.6.25.16" target="_blank">2.6.25.16</a>. After similar problems with 1.3.x we installed 1.4.0qa61 and set up a /scratch for testing using the following script "glusterconf.sh" which runs local on all nodes on startup and writes the two config files /usr/local/etc/glusterfs-{server,client}.vol:<br>
<br>
---------------------------------- 8< snip >8 ----------------------------------<br>
#!/bin/sh<br>
<br>
HOST=$(hostname -s)<br>
<br>
if [ $HOST = master ];then<br>
MASTER_IP=<a href="http://127.0.0.1" target="_blank">127.0.0.1</a><br>
HOST_IP=<a href="http://127.0.0.1" target="_blank">127.0.0.1</a><br>
HOST_N=0<br>
else<br>
MASTER_IP=<a href="http://192.168.1.254" target="_blank">192.168.1.254</a><br>
HOST_IP=$(hostname -i)<br>
HOST_N=${HOST_IP##*.}<br>
fi<br>
<br>
LOCAL=sc$HOST_N<br>
<br>
###################################################################<br>
# write /usr/local/etc/glusterfs-server.vol<br>
{<br>
<br>
cat <<EOF<br>
###<br>
### Server config automatically created by $PWD/$0<br>
###<br>
<br>
EOF<br>
<br>
if [ $HOST = master ];then<br>
SERVERVOLUMES="scns"<br>
cat <<EOF<br>
volume scns<br>
type storage/posix<br>
option directory /export/scratch_ns<br>
end-volume<br>
<br>
EOF<br>
else # if master<br>
SERVERVOLUMES=""<br>
fi # if master<br>
<br>
SERVERVOLUMES="$SERVERVOLUMES $LOCAL"<br>
cat <<EOF<br>
volume $LOCAL-posix<br>
type storage/posix<br>
option directory /export/scratch<br>
end-volume<br>
<br>
volume $LOCAL-locks<br>
type features/posix-locks<br>
subvolumes $LOCAL-posix<br>
end-volume<br>
<br>
volume $LOCAL-ioth<br>
type performance/io-threads<br>
option thread-count 4<br>
subvolumes $LOCAL-locks<br>
end-volume<br>
<br>
volume $LOCAL<br>
type performance/read-ahead<br>
subvolumes $LOCAL-ioth<br>
end-volume<br>
<br>
volume server<br>
type protocol/server<br>
option transport-type tcp/server<br>
subvolumes $SERVERVOLUMES<br>
EOF<br>
<br>
for vol in $SERVERVOLUMES;do<br>
cat <<EOF<br>
option auth.addr.$vol.allow <a href="http://127.0.0.1" target="_blank">127.0.0.1</a>,<a href="http://192.168.1." target="_blank">192.168.1.</a>*<br>
EOF<br>
done<br>
<br>
cat <<EOF<br>
end-volume<br>
<br>
EOF<br>
<br>
} > /usr/local/etc/glusterfs-server.vol<br>
<br>
###################################################################<br>
# write /usr/local/etc/glusterfs-client.vol<br>
{<br>
cat <<EOF<br>
###<br>
### Client config automatically created by $PWD/$0<br>
###<br>
<br>
volume scns<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host $MASTER_IP<br>
option remote-subvolume scns<br>
end-volume<br>
<br>
volume sc0<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host $MASTER_IP<br>
option remote-subvolume sc0<br>
end-volume<br>
<br>
EOF<br>
<br>
UNIFY="sc0"<br>
<br>
# leave out node66 at the moment...<br>
<br>
for n in $(seq 65) $(seq 67 87);do<br>
VOL=sc$n<br>
UNIFY="$UNIFY $VOL"<br>
cat <<EOF<br>
volume $VOL<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://192.168.1." target="_blank">192.168.1.</a>$n<br>
option remote-subvolume $VOL<br>
end-volume<br>
<br>
EOF<br>
done<br>
<br>
cat <<EOF<br>
volume scratch<br>
type cluster/unify<br>
subvolumes $UNIFY<br>
option namespace scns<br>
option scheduler nufa<br>
option nufa.limits.min-free-disk 15<br>
option nufa.refresh-interval 10<br>
option nufa.local-volume-name $LOCAL<br>
end-volume<br>
<br>
volume scratch-io-threads<br>
type performance/io-threads<br>
option thread-count 4<br>
subvolumes scratch<br>
end-volume<br>
<br>
volume scratch-write-behind<br>
type performance/write-behind<br>
option aggregate-size 128kB<br>
option flush-behind off<br>
subvolumes scratch-io-threads<br>
end-volume<br>
<br>
volume scratch-read-ahead<br>
type performance/read-ahead<br>
option page-size 128kB # unit in bytes<br>
option page-count 2 # cache per file = (page-count x page-size)<br>
subvolumes scratch-write-behind<br>
end-volume<br>
<br>
volume scratch-io-cache<br>
type performance/io-cache<br>
option cache-size 64MB<br>
option page-size 512kB<br>
subvolumes scratch-read-ahead<br>
end-volume<br>
<br>
EOF<br>
<br>
} > /usr/local/etc/glusterfs-client.vol<br>
---------------------------------- 8< snip >8 ----------------------------------<br>
<br>
The cluster uses MPI over Infiniband, while GlusterFS runs over TCP/IP Gigabit Ethernet. I use FUSE 2.7.4 with patch fuse-2.7.3glfs10.diff (Is that OK? The patch succeeded)<br>
<br>
Everything is fine until some nodes which are used by a job block on access to /scratch or, sometimes later, give<br>
<br>
df: `/scratch': Transport endpoint is not connected<br>
<br>
The glusterfs.log on node36 is flooded by<br>
<br>
2008-11-25 07:30:35 E [client-protocol.c:243:call_bail] sc70: activating bail-out. pending frames = 3. last sent = 2008-11-25 07:29:52. last received = 2008-11-25 07:29:49. transport-timeout = 42<br>
2008-11-25 07:30:35 C [client-protocol.c:250:call_bail] sc70: bailing transport<br>
...(~100MB)<br>
<br>
(~2 lines for every node every 10 seconds) Furthermore, I find at the end of glusterfs.log:<br>
<br>
grep -v call_bail glusterfs.log<br>
...<br>
2008-11-25 10:00:46 E [socket.c:1187:socket_submit] sc0: transport not connected to submit (priv->connected = 255)<br>
...<br>
2008-11-25 10:00:46 E [socket.c:1187:socket_submit] sc87: transport not connected to submit (priv->connected = 255)<br>
2008-11-25 10:00:46 E [socket.c:1187:socket_submit] scns: transport not connected to submit (priv->connected = 255)<br>
2008-11-25 10:05:03 E [fuse-bridge.c:1886:fuse_statfs_cbk] glusterfs-fuse: 1353: ERR => -1 (Transport endpoint is not connected)<br>
<br>
On node68 I find<br>
<br>
2008-11-24 23:20:12 W [client-protocol.c:93:this_ino_set] sc0: inode number(201326854) changed for inode(0x6130d0)<br>
2008-11-24 23:20:12 W [client-protocol.c:93:this_ino_set] scns: inode number(37749030) changed for inode(0x6130d0)<br>
2008-11-24 23:20:58 E [client-protocol.c:243:call_bail] scns: activating bail-out. pending frames = 3. last sent = 2008-11-24 23:20:12. last received = 2008-11-24 23:20:12. transport-timeout = 42<br>
2008-11-24 23:20:58 C [client-protocol.c:250:call_bail] scns: bailing transport<br>
2008-11-24 23:20:58 E [client-protocol.c:243:call_bail] sc0: activating bail-out. pending frames = 3. last sent = 2008-11-24 23:20:12. last received = 2008-11-24 23:20:12. transport-timeout = 42<br>
2008-11-24 23:20:58 C [client-protocol.c:250:call_bail] sc0: bailing transport<br>
...(~100MB)<br>
<br>
only for scns and sc0 and then<br>
<br>
2008-11-25 10:01:31 E [client-protocol.c:243:call_bail] sc1: activating bail-out. pending frames = 1. last sent = 2008-11-25 10:00:46. last received = 2008-11-24 23:20:12. transport-timeout = 42<br>
2008-11-25 10:01:31 C [client-protocol.c:250:call_bail] sc1: bailing transport<br>
...(~100MB)<br>
<br>
for all nodes, as well as<br>
<br>
2008-11-25 10:00:46 E [socket.c:1187:socket_submit] sc0: transport not connected to submit (priv->connected = 255)<br>
2008-11-25 10:00:46 E [socket.c:1187:socket_submit] scns: transport not connected to submit (priv->connected = 255)<br>
2008-11-25 11:23:18 E [socket.c:1187:socket_submit] sc1: transport not connected to submit (priv->connected = 255)<br>
2008-11-25 11:23:18 E [socket.c:1187:socket_submit] sc2: transport not connected to submit (priv->connected = 255)<br>
...<br>
<br>
The third affected node node77 says:<br>
<br>
2008-11-24 22:07:20 W [client-protocol.c:93:this_ino_set] sc0: inode number(201326854) changed for inode(0x7f97d6c0ac70)<br>
2008-11-24 22:07:20 W [client-protocol.c:93:this_ino_set] scns: inode number(37749030) changed for inode(0x7f97d6c0ac70)<br>
2008-11-24 22:08:07 E [client-protocol.c:243:call_bail] sc10: activating bail-out. pending frames = 7. last sent = 2008-11-24 22:07:24. last received = 2008-11-24 22:07:20. transport-timeout = 42<br>
2008-11-24 22:08:07 C [client-protocol.c:250:call_bail] sc10: bailing transport<br>
...(~100MB)<br>
<br>
and then<br>
<br>
2008-11-25 10:00:46 E [socket.c:1187:socket_submit] sc0: transport not connected to submit (priv->connected = 255)<br>
...<br>
2008-11-25 10:00:46 E [socket.c:1187:socket_submit] sc87: transport not connected to submit (priv->connected = 255)<br>
2008-11-25 10:00:46 E [socket.c:1187:socket_submit] scns: transport not connected to submit (priv->connected = 255)<br>
<br>
<br>
As I said, similar problems occurred with version 1.3.x. If these problems cannot be solved, we have to use a different file system, so any help is very appreciated.<br>
<br>
Have fun,<br>
<br>
Fred<br>
<br>
Dr. Fred Hucht <<a href="mailto:fred@thp.Uni-DuE.de" target="_blank">fred@thp.Uni-DuE.de</a>><br>
Institute for Theoretical Physics<br>
University of Duisburg-Essen, 47048 Duisburg, Germany<br>
<br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div><br><br clear="all"><br>-- <br>hard work often pays off after time, but laziness always pays off now<br>