Hi,<div><br></div><div>Last night, we got some troubles with a GlusterFS mount. It's a replicate volume, and the 10.1.1.2 host was already down. The volume files weren't readable until I manually restarted the GlusterFS instance.</div>
<div>We'd like to understand what happened on this volume. Especially the "Server <a href="http://10.1.1.1:6996">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting." message. I can't figure out why the GlusterFS instance couldn't talk to itself.</div>
<div>Please help us.</div><div><br></div><div>This log is from 10.1.1.1 itself :</div><div><br></div><div><div>[2010-06-01 00:01:54] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2010-06-01 00:04:28] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div><div>[2010-06-01 00:06:57] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2010-06-01 00:09:32] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div><div>[2010-06-01 00:11:55] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2010-06-01 00:14:29] E [client-protocol.c:415:client_ping_timer_expired] brick-qmaster: Server <a href="http://10.1.1.1:6996">10.1.1.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame STAT(0) frame sent = 2010-05-31 23:45:43. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731899: STAT() /masterspool => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame LOOKUP(27) frame sent = 2010-05-31 23:45:39. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731898: LOOKUP() / => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame STATFS(13) frame sent = 2010-05-31 23:45:39. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:2352:fuse_statfs_cbk] glusterfs-fuse: 7731897: ERR => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame LOOKUP(27) frame sent = 2010-05-31 23:45:37. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731896: LOOKUP() / => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame OPEN(10) frame sent = 2010-05-31 23:45:34. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:858:fuse_fd_cbk] glusterfs-fuse: 7731894: OPEN() /cell/common/bootstrap => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame FSTAT(25) frame sent = 2010-05-31 23:45:35. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731895: FSTAT() /masterspool/messages => -1 (File descriptor in bad state)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame FSTAT(25) frame sent = 2010-05-31 23:45:34. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:44] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731893: FSTAT() /cell/common/bootstrap => -1 (File descriptor in bad state)</div><div>[2010-06-01 00:15:44] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame PING(5) frame sent = 2010-05-31 23:45:35. frame-timeout = 1800</div>
<div>[2010-06-01 00:15:54] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame PING(5) frame sent = 2010-05-31 23:45:51. frame-timeout = 1800</div><div>[2010-06-01 00:16:05] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame LOOKUP(27) frame sent = 2010-05-31 23:45:56. frame-timeout = 1800</div>
<div>[2010-06-01 00:16:05] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7731901: LOOKUP() / => -1 (Transport endpoint is not connected)</div><div>[2010-06-01 00:16:25] E [client-protocol.c:313:call_bail] brick-qmaster: bailing out frame STATFS(13) frame sent = 2010-05-31 23:46:19. frame-timeout = 1800</div>
<div>[2010-06-01 00:16:25] W [fuse-bridge.c:2352:fuse_statfs_cbk] glusterfs-fuse: 7731902: ERR => -1 (Transport endpoint is not connected)</div><div>[..]</div><div><br></div><div>Here is our configuration :</div><div>
<div>
<br></div><div>volume posix</div><div> type storage/posix</div><div> option directory /data/sge</div><div>end-volume</div><div><br></div><div>volume locks</div><div> type features/locks</div><div> subvolumes posix</div>
<div>end-volume</div><div><br></div><div>volume brick</div><div> type performance/io-threads</div><div> option thread-count 8</div><div> subvolumes locks</div><div>end-volume</div><div><br></div><div>volume server</div>
<div> type protocol/server</div><div> option transport-type tcp</div><div> option auth.addr.brick.allow 10.*.*.*</div><div> subvolumes brick</div><div>end-volume</div><div><br></div><div>volume brick-qmaster</div>
<div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.1.1.1</div><div> option remote-subvolume brick</div><div>end-volume</div><div><br></div><div>volume brick-shadow</div>
<div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.1.1.2</div><div> option remote-subvolume brick</div><div>end-volume</div><div><br></div><div>volume sge-replicate</div>
<div> type cluster/replicate</div><div> subvolumes brick-qmaster brick-shadow</div><div>end-volume</div></div><div><br></div><div><br></div><div><br></div>Philippe Muller<br>
</div>