<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: Times New Roman; font-size: 12pt; color: #000000'><span>The engineering team will need some details - <br><br>Gluster version?<br>OS details for the clients and servers.<br>Hardware</span><span id="cbc0a866-40d6-4bea-b2eb-024073fd2a3b"> details for the clients and servers</span><br><span id="cbc0a866-40d6-4bea-b2eb-024073fd2a3b">Client volume file.<br>Why was 10.1.1.2 already down, how was it brought down?<br><br>Also, this type of question will probably get a better response on the gluster-users list, could you subscribe there and repost your email with the details I've asked for? You can subscribe to Gluster-users here - <a mce_href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a>.<br><br><div><div>Thanks, <br>
<br>
Craig<br>
<br>--<br>Craig
Carl<br><div><div><div><div><div><div><div><div><div><div><div><div><div>Sales
Engineer; Gluster, Inc. </div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></span><br><hr><b>From: </b>"Philippe Muller" <philippe.muller@gmail.com><br><b>To: </b>gluster-devel@nongnu.org<br><b>Sent: </b>Tuesday, June 1, 2010 1:39:09 AM<br><b>Subject: </b>[Gluster-devel] GlusterFS Volume Failure<br><br>Hi,<div><br></div><div>Last night, we got some troubles with a GlusterFS mount. It's a replicate volume, and the 10.1.1.2 host was already down. The volume files weren't readable until I manually restarted the GlusterFS instance.</div>
<div>We'd like to understand what happened on this volume. Especially the "Server <a href="http://10.1.1.1:6996" mce_href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting." message. I can't figure out why the GlusterFS instance couldn't talk to itself.</div>
<div>Please help us.</div><div><br></div><div>This log is from 10.1.1.1 itself :</div><div><br></div><div><div>[2010-06-01 00:01:54] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996" mce_href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2010-06-01 00:04:28] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996" mce_href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div><div>[2010-06-01 00:06:57] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996" mce_href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2010-06-01 00:09:32] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996" mce_href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div><div>[2010-06-01 00:11:55] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996" mce_href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2010-06-01 00:14:29] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996" mce_href="http://10.1.1.1:6996" target="_blank">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame STAT(0) frame sent = 2010-05-31 23:45:43. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731899: STAT() /masterspool => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame LOOKUP(27) frame sent = 2010-05-31 23:45:39. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731898: LOOKUP() / => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame STATFS(13) frame sent = 2010-05-31 23:45:39. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:2352:fuse_statfs_cbk] glusterfs-fuse: 7731897: ERR => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame LOOKUP(27) frame sent = 2010-05-31 23:45:37. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731896: LOOKUP() / => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame OPEN(10) frame sent = 2010-05-31 23:45:34. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:858:fuse_fd_cbk] glusterfs-fuse: 7731894: OPEN() /cell/common/bootstrap => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame FSTAT(25) frame sent = 2010-05-31 23:45:35. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731895: FSTAT() /masterspool/messages => -1 (File descriptor in bad state)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame FSTAT(25) frame sent = 2010-05-31 23:45:34. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731893: FSTAT() /cell/common/bootstrap => -1 (File descriptor in bad state)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame PING(5) frame sent = 2010-05-31 23:45:35. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:54] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame PING(5) frame sent = 2010-05-31 23:45:51. frame-timeout = 1800</div><div>[2010-06-01 00:16:05] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame LOOKUP(27) frame sent = 2010-05-31 23:45:56. frame-timeout = 1800</div>
<div>[2010-06-01 00:16:05] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731901: LOOKUP() / => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:16:25] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame STATFS(13) frame sent = 2010-05-31 23:46:19. frame-timeout = 1800</div>
<div>[2010-06-01 00:16:25] W [fuse-bridge.c:2352:fuse_statfs_cbk] glusterfs-fuse: 7731902: ERR => -1 (Transport endpoint is not connected)</div><div>[..]</div><div><br></div><div>Here is our configuration :</div><div>
<div>
<br></div><div>volume posix</div><div> type storage/posix</div><div> option directory /data/sge</div><div>end-volume</div><div><br></div><div>volume locks</div><div> type features/locks</div><div> subvolumes posix</div>
<div>end-volume</div><div><br></div><div>volume brick</div><div> type performance/io-threads</div><div> option thread-count 8</div><div> subvolumes locks</div><div>end-volume</div><div><br></div><div>volume server</div>
<div> type protocol/server</div><div> option transport-type tcp</div><div> option auth.addr.brick.allow 10.*.*.*</div><div> subvolumes brick</div><div>end-volume</div><div><br></div><div>volume brick-qmaster</div>
<div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.1.1.1</div><div> option remote-subvolume brick</div><div>end-volume</div><div><br></div><div>volume brick-shadow</div>
<div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.1.1.2</div><div> option remote-subvolume brick</div><div>end-volume</div><div><br></div><div>volume sge-replicate</div>
<div> type cluster/replicate</div><div> subvolumes brick-qmaster brick-shadow</div><div>end-volume</div></div><div><br></div><div><br></div><div><br></div>Philippe Muller<br>
</div>
<br>_______________________________________________<br>Gluster-devel mailing list<br>Gluster-devel@nongnu.org<br>http://lists.nongnu.org/mailman/listinfo/gluster-devel<br></div></body></html>