<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Hi!</div><div><br></div><div>The glusterfsd.log on all nodes are virtually empty, the only entry on 2008-11-25 reads</div><div><br></div><div>2008-11-25 03:13:48 E [io-threads.c:273:iot_flush] sc1-ioth: fd context is NULL, returning EBADFD</div><div><br></div><div>on all nodes. I don't think that this is related to our problems.</div><div><br></div><div>Regards,</div><div><br></div><div> Fred</div><div><br></div><div><div>On 25.11.2008, at 13:17, Basavanagowda Kanur wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Fred,<br> Can you also provide us server logs?<br> <br>--<br>gowda<br><br><br><div class="gmail_quote">On Tue, Nov 25, 2008 at 4:57 PM, Fred Hucht <span dir="ltr"><<a href="mailto:fred@thp.uni-due.de">fred@thp.uni-due.de</a>></span> wrote:<br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi devels!<br> <br> We consider GlusterFS as parallel file server (8 server nodes) for our parallel Opteron cluster (88 nodes, ~500 cores), as well as for a unified nufa /scratch distributed over all nodes. We use the cluster within a scientific environment (theoretical physics) and use Scientific Linux with kernel <a href="http://2.6.25.16" target="_blank">2.6.25.16</a>. After similar problems with 1.3.x we installed 1.4.0qa61 and set up a /scratch for testing using the following script "glusterconf.sh" which runs local on all nodes on startup and writes the two config files /usr/local/etc/glusterfs-{server,client}.vol:<br> <br> ---------------------------------- 8< snip >8 ----------------------------------<br> #!/bin/sh<br> <br> HOST=$(hostname -s)<br> <br> if [ $HOST = master ];then<br> MASTER_IP=<a href="http://127.0.0.1" target="_blank">127.0.0.1</a><br> HOST_IP=<a href="http://127.0.0.1" target="_blank">127.0.0.1</a><br> HOST_N=0<br> else<br> MASTER_IP=<a href="http://192.168.1.254" target="_blank">192.168.1.254</a><br> HOST_IP=$(hostname -i)<br> HOST_N=${HOST_IP##*.}<br> fi<br> <br> LOCAL=sc$HOST_N<br> <br> ###################################################################<br> # write /usr/local/etc/glusterfs-server.vol<br> {<br> <br> cat <<EOF<br> ###<br> ### Server config automatically created by $PWD/$0<br> ###<br> <br> EOF<br> <br> if [ $HOST = master ];then<br> SERVERVOLUMES="scns"<br> cat <<EOF<br> volume scns<br> type storage/posix<br> option directory /export/scratch_ns<br> end-volume<br> <br> EOF<br> else # if master<br> SERVERVOLUMES=""<br> fi # if master<br> <br> SERVERVOLUMES="$SERVERVOLUMES $LOCAL"<br> cat <<EOF<br> volume $LOCAL-posix<br> type storage/posix<br> option directory /export/scratch<br> end-volume<br> <br> volume $LOCAL-locks<br> type features/posix-locks<br> subvolumes $LOCAL-posix<br> end-volume<br> <br> volume $LOCAL-ioth<br> type performance/io-threads<br> option thread-count 4<br> subvolumes $LOCAL-locks<br> end-volume<br> <br> volume $LOCAL<br> type performance/read-ahead<br> subvolumes $LOCAL-ioth<br> end-volume<br> <br> volume server<br> type protocol/server<br> option transport-type tcp/server<br> subvolumes $SERVERVOLUMES<br> EOF<br> <br> for vol in $SERVERVOLUMES;do<br> cat <<EOF<br> option auth.addr.$vol.allow <a href="http://127.0.0.1" target="_blank">127.0.0.1</a>,<a href="http://192.168.1." target="_blank">192.168.1.</a>*<br> EOF<br> done<br> <br> cat <<EOF<br> end-volume<br> <br> EOF<br> <br> } > /usr/local/etc/glusterfs-server.vol<br> <br> ###################################################################<br> # write /usr/local/etc/glusterfs-client.vol<br> {<br> cat <<EOF<br> ###<br> ### Client config automatically created by $PWD/$0<br> ###<br> <br> volume scns<br> type protocol/client<br> option transport-type tcp/client<br> option remote-host $MASTER_IP<br> option remote-subvolume scns<br> end-volume<br> <br> volume sc0<br> type protocol/client<br> option transport-type tcp/client<br> option remote-host $MASTER_IP<br> option remote-subvolume sc0<br> end-volume<br> <br> EOF<br> <br> UNIFY="sc0"<br> <br> # leave out node66 at the moment...<br> <br> for n in $(seq 65) $(seq 67 87);do<br> VOL=sc$n<br> UNIFY="$UNIFY $VOL"<br> cat <<EOF<br> volume $VOL<br> type protocol/client<br> option transport-type tcp/client<br> option remote-host <a href="http://192.168.1." target="_blank">192.168.1.</a>$n<br> option remote-subvolume $VOL<br> end-volume<br> <br> EOF<br> done<br> <br> cat <<EOF<br> volume scratch<br> type cluster/unify<br> subvolumes $UNIFY<br> option namespace scns<br> option scheduler nufa<br> option nufa.limits.min-free-disk 15<br> option nufa.refresh-interval 10<br> option nufa.local-volume-name $LOCAL<br> end-volume<br> <br> volume scratch-io-threads<br> type performance/io-threads<br> option thread-count 4<br> subvolumes scratch<br> end-volume<br> <br> volume scratch-write-behind<br> type performance/write-behind<br> option aggregate-size 128kB<br> option flush-behind off<br> subvolumes scratch-io-threads<br> end-volume<br> <br> volume scratch-read-ahead<br> type performance/read-ahead<br> option page-size 128kB # unit in bytes<br> option page-count 2 # cache per file = (page-count x page-size)<br> subvolumes scratch-write-behind<br> end-volume<br> <br> volume scratch-io-cache<br> type performance/io-cache<br> option cache-size 64MB<br> option page-size 512kB<br> subvolumes scratch-read-ahead<br> end-volume<br> <br> EOF<br> <br> } > /usr/local/etc/glusterfs-client.vol<br> ---------------------------------- 8< snip >8 ----------------------------------<br> <br> The cluster uses MPI over Infiniband, while GlusterFS runs over TCP/IP Gigabit Ethernet. I use FUSE 2.7.4 with patch fuse-2.7.3glfs10.diff (Is that OK? The patch succeeded)<br> <br> Everything is fine until some nodes which are used by a job block on access to /scratch or, sometimes later, give<br> <br> df: `/scratch': Transport endpoint is not connected<br> <br> The glusterfs.log on node36 is flooded by<br> <br> 2008-11-25 07:30:35 E [client-protocol.c:243:call_bail] sc70: activating bail-out. pending frames = 3. last sent = 2008-11-25 07:29:52. last received = 2008-11-25 07:29:49. transport-timeout = 42<br> 2008-11-25 07:30:35 C [client-protocol.c:250:call_bail] sc70: bailing transport<br> ...(~100MB)<br> <br> (~2 lines for every node every 10 seconds) Furthermore, I find at the end of glusterfs.log:<br> <br> grep -v call_bail glusterfs.log<br> ...<br> 2008-11-25 10:00:46 E [socket.c:1187:socket_submit] sc0: transport not connected to submit (priv->connected = 255)<br> ...<br> 2008-11-25 10:00:46 E [socket.c:1187:socket_submit] sc87: transport not connected to submit (priv->connected = 255)<br> 2008-11-25 10:00:46 E [socket.c:1187:socket_submit] scns: transport not connected to submit (priv->connected = 255)<br> 2008-11-25 10:05:03 E [fuse-bridge.c:1886:fuse_statfs_cbk] glusterfs-fuse: 1353: ERR => -1 (Transport endpoint is not connected)<br> <br> On node68 I find<br> <br> 2008-11-24 23:20:12 W [client-protocol.c:93:this_ino_set] sc0: inode number(201326854) changed for inode(0x6130d0)<br> 2008-11-24 23:20:12 W [client-protocol.c:93:this_ino_set] scns: inode number(37749030) changed for inode(0x6130d0)<br> 2008-11-24 23:20:58 E [client-protocol.c:243:call_bail] scns: activating bail-out. pending frames = 3. last sent = 2008-11-24 23:20:12. last received = 2008-11-24 23:20:12. transport-timeout = 42<br> 2008-11-24 23:20:58 C [client-protocol.c:250:call_bail] scns: bailing transport<br> 2008-11-24 23:20:58 E [client-protocol.c:243:call_bail] sc0: activating bail-out. pending frames = 3. last sent = 2008-11-24 23:20:12. last received = 2008-11-24 23:20:12. transport-timeout = 42<br> 2008-11-24 23:20:58 C [client-protocol.c:250:call_bail] sc0: bailing transport<br> ...(~100MB)<br> <br> only for scns and sc0 and then<br> <br> 2008-11-25 10:01:31 E [client-protocol.c:243:call_bail] sc1: activating bail-out. pending frames = 1. last sent = 2008-11-25 10:00:46. last received = 2008-11-24 23:20:12. transport-timeout = 42<br> 2008-11-25 10:01:31 C [client-protocol.c:250:call_bail] sc1: bailing transport<br> ...(~100MB)<br> <br> for all nodes, as well as<br> <br> 2008-11-25 10:00:46 E [socket.c:1187:socket_submit] sc0: transport not connected to submit (priv->connected = 255)<br> 2008-11-25 10:00:46 E [socket.c:1187:socket_submit] scns: transport not connected to submit (priv->connected = 255)<br> 2008-11-25 11:23:18 E [socket.c:1187:socket_submit] sc1: transport not connected to submit (priv->connected = 255)<br> 2008-11-25 11:23:18 E [socket.c:1187:socket_submit] sc2: transport not connected to submit (priv->connected = 255)<br> ...<br> <br> The third affected node node77 says:<br> <br> 2008-11-24 22:07:20 W [client-protocol.c:93:this_ino_set] sc0: inode number(201326854) changed for inode(0x7f97d6c0ac70)<br> 2008-11-24 22:07:20 W [client-protocol.c:93:this_ino_set] scns: inode number(37749030) changed for inode(0x7f97d6c0ac70)<br> 2008-11-24 22:08:07 E [client-protocol.c:243:call_bail] sc10: activating bail-out. pending frames = 7. last sent = 2008-11-24 22:07:24. last received = 2008-11-24 22:07:20. transport-timeout = 42<br> 2008-11-24 22:08:07 C [client-protocol.c:250:call_bail] sc10: bailing transport<br> ...(~100MB)<br> <br> and then<br> <br> 2008-11-25 10:00:46 E [socket.c:1187:socket_submit] sc0: transport not connected to submit (priv->connected = 255)<br> ...<br> 2008-11-25 10:00:46 E [socket.c:1187:socket_submit] sc87: transport not connected to submit (priv->connected = 255)<br> 2008-11-25 10:00:46 E [socket.c:1187:socket_submit] scns: transport not connected to submit (priv->connected = 255)<br> <br> <br> As I said, similar problems occurred with version 1.3.x. If these problems cannot be solved, we have to use a different file system, so any help is very appreciated.<br> <br> Have fun,<br> <br> Fred<br> <br> Dr. Fred Hucht <<a href="mailto:fred@thp.Uni-DuE.de" target="_blank">fred@thp.Uni-DuE.de</a>><br> Institute for Theoretical Physics<br> University of Duisburg-Essen, 47048 Duisburg, Germany<br> <br> <br> <br> _______________________________________________<br> Gluster-devel mailing list<br> <a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br> <a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br> </blockquote></div><br><br clear="all"><br>-- <br>hard work often pays off after time, but laziness always pays off now<br></blockquote></div><br><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Dr. Fred Hucht <<a href="mailto:fred@thp.Uni-DuE.de">fred@thp.Uni-DuE.de</a>></div><div><div>Institute for Theoretical Physics</div><div>University of Duisburg-Essen, 47048 Duisburg, Germany</div></div></div></span></div></span></span></div></span></div></span></div></span></div></span> </div><br></body></html>