<div class="gmail_quote">Sorry, I forgot to include these information :<div>- The configuration I gave you contains the server and the client part, as used on the 10.1.1.1 host (and 10.1.1.2)</div><div> - The instance mounts itself ; we run it this way : /usr/local/sbin/glusterfs --log-file=/var/log/glusterfs/sge.log --volfile=/usr/local/etc/glusterfs/sge.vol --pid-file=/var/run/glusterfs-sge.pid /mnt/sge</div>
<div>- The clients use the client part of this configuration<br><div><div>- Linux kernel : 2.6.32.7</div><div>- GlusterFS version : 3.0.3 (built from sources)</div><div>- Volume role : share configuration and master spool data between GridEngine masters and clients (master writes data / clients read data)</div>
<div><br></div><div>Originally, we had two hosts replicating the volume data (10.1.1.1 and 10.1.1.2). Last week, we had to change the second host IP address.</div><div>When we updated clients configurations to use the new address, we got a lot of "I/O error" when reading files. Since this volume is critical for us, we chose to lose the redundancy and quickly get the service back to normal (= 10.1.1.2 shutdown).</div>
<div>Then, we got the incident described in the last e-mail.</div><div>At this time we didn't lost any data. However, regarding these events, I'm somewhat afraid of losing some data.</div><div><br></div><div>Does someone already used GlusterFS to store GridEngine configuration/master spool data ?</div>
<div><br></div><div>For reference, the client volume :</div><div><br></div><div><div class="im"><div>volume brick-qmaster</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.1.1.1</div>
<div> option remote-subvolume brick</div><div>end-volume</div><div>volume brick-shadow</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.1.1.1</div><div> option remote-subvolume brick</div>
<div>end-volume</div></div><div class="im"><div>volume sge-replicate</div><div> type cluster/replicate</div><div> subvolumes brick-qmaster brick-shadow</div><div>end-volume</div></div></div><div><br></div><div><br>
</div><div>Regards,</div><div>
<br clear="all"><font color="#888888">Philippe Muller</font><div><div></div><div class="h5"><br>
<br><br><div class="gmail_quote">On Tue, Jun 1, 2010 at 10:51 AM, Craig Carl <span dir="ltr"><<a href="mailto:craig@gluster.com" target="_blank">craig@gluster.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div style="font-family:Times New Roman;font-size:12pt;color:#000000"><span>The engineering team will need some details - <br><br>Gluster version?<br>OS details for the clients and servers.<br>Hardware</span><span> details for the clients and servers</span><br>
<span>Client volume file.<br>Why was 10.1.1.2 already down, how was it brought down?<br><br>Also, this type of question will probably get a better response on the gluster-users list, could you subscribe there and repost your email with the details I've asked for? You can subscribe to Gluster-users here - <a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a>.<br>
<br><div><div>Thanks, <br>
<br>
Craig<br>
<br>--<br>Craig
Carl<br><div><div><div><div><div><div><div><div><div><div><div><div><div>Sales
Engineer; Gluster, Inc. </div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></span><br><hr><b>From: </b>"Philippe Muller" <<a href="mailto:philippe.muller@gmail.com" target="_blank">philippe.muller@gmail.com</a>><br>
<b>To: </b><a href="mailto:gluster-devel@nongnu.org" target="_blank">gluster-devel@nongnu.org</a><br><b>Sent: </b>Tuesday, June 1, 2010 1:39:09 AM<br><b>Subject: </b>[Gluster-devel] GlusterFS Volume Failure<div><div></div>
<div><br><br>Hi,<div><br></div><div>Last night, we got some troubles with a GlusterFS mount. It's a replicate volume, and the 10.1.1.2 host was already down. The volume files weren't readable until I manually restarted the GlusterFS instance.</div>
<div>We'd like to understand what happened on this volume. Especially the "Server <a href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting." message. I can't figure out why the GlusterFS instance couldn't talk to itself.</div>
<div>Please help us.</div><div><br></div><div>This log is from 10.1.1.1 itself :</div><div><br></div><div><div>[2010-06-01 00:01:54] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2010-06-01 00:04:28] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2010-06-01 00:06:57] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2010-06-01 00:09:32] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2010-06-01 00:11:55] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2010-06-01 00:14:29] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame STAT(0) frame sent = 2010-05-31 23:45:43. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731899: STAT() /masterspool => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame LOOKUP(27) frame sent = 2010-05-31 23:45:39. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731898: LOOKUP() / => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame STATFS(13) frame sent = 2010-05-31 23:45:39. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:2352:fuse_statfs_cbk] glusterfs-fuse: 7731897: ERR => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame LOOKUP(27) frame sent = 2010-05-31 23:45:37. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731896: LOOKUP() / => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame OPEN(10) frame sent = 2010-05-31 23:45:34. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:858:fuse_fd_cbk] glusterfs-fuse: 7731894: OPEN() /cell/common/bootstrap => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame FSTAT(25) frame sent = 2010-05-31 23:45:35. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731895: FSTAT() /masterspool/messages => -1 (File descriptor in bad state)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame FSTAT(25) frame sent = 2010-05-31 23:45:34. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731893: FSTAT() /cell/common/bootstrap => -1 (File descriptor in bad state)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame PING(5) frame sent = 2010-05-31 23:45:35. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:54] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame PING(5) frame sent = 2010-05-31 23:45:51. frame-timeout = 1800</div><div>[2010-06-01 00:16:05] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame LOOKUP(27) frame sent = 2010-05-31 23:45:56. frame-timeout = 1800</div>
<div>[2010-06-01 00:16:05] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731901: LOOKUP() / => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:16:25] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame STATFS(13) frame sent = 2010-05-31 23:46:19. frame-timeout = 1800</div>
<div>[2010-06-01 00:16:25] W [fuse-bridge.c:2352:fuse_statfs_cbk] glusterfs-fuse: 7731902: ERR => -1 (Transport endpoint is not connected)</div><div>[..]</div><div><br></div><div>Here is our configuration :</div><div>
<div>
<br></div><div>volume posix</div><div> type storage/posix</div><div> option directory /data/sge</div><div>end-volume</div><div><br></div><div>volume locks</div><div> type features/locks</div><div> subvolumes posix</div>
<div>end-volume</div><div><br></div><div>volume brick</div><div> type performance/io-threads</div><div> option thread-count 8</div><div> subvolumes locks</div><div>end-volume</div><div><br></div><div>volume server</div>
<div> type protocol/server</div><div> option transport-type tcp</div><div> option auth.addr.brick.allow 10.*.*.*</div><div> subvolumes brick</div><div>end-volume</div><div><br></div><div>volume brick-qmaster</div>
<div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.1.1.1</div><div> option remote-subvolume brick</div><div>end-volume</div><div><br></div><div>volume brick-shadow</div>
<div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.1.1.2</div><div> option remote-subvolume brick</div><div>end-volume</div><div><br></div><div>volume sge-replicate</div>
<div> type cluster/replicate</div><div> subvolumes brick-qmaster brick-shadow</div><div>end-volume</div></div><div><br></div><div><br></div><div><br></div>Philippe Muller<br>
</div>
<br></div></div>_______________________________________________<br>Gluster-devel mailing list<br><a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br><a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br></div></div></div></div></div>
</div><br>