Hi Roland,<br><br>Can you attach gdb to servers and get the backtrace?<br><br>regards,<br><div class="gmail_quote">On Fri, Dec 18, 2009 at 3:16 PM, Roland Fischer <span dir="ltr"><<a href="mailto:roland.fischer@xidras.com">roland.fischer@xidras.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">hi all,<br>
<br>
glusterfsversion 3.0.0<br>
replication: server-side-replication<br>
<br>
we have 2 glusterfs server with glusterfs v 3.0.0 and one clienthost with glusterfs v3.0.0. Now we have one partition which contains office-files and short videos.<br>
<br>
We simulate a serverfailture on gfs-01-01 (rcnetwork stop; sleep 300; rcnetwork start). After rcnetwork stop on gfs-01-01 we see at the client host at mountpoints<br>
<br>
/GFS/office-data02': Transport endpoint is not connected => we cant do anything on this partition. If gfs-01-01 comes back online the partition is accessable. same situation at gfs-01-02<br>
but here if we make a df the clienthost does nothing. if gfs-01-02 is back online the partition is again accessable.<br>
<br>
clientlog. (rcnetwork down on gfs-01-01)<br>
[2009-12-18 12:06:51] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 41: STAT() /funny => -1 (Transport endpoint is not connected)<br>
[2009-12-18 12:06:55] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 42: STAT() /funny => -1 (Transport endpoint is not connected)<br>
[2009-12-18 12:06:55] W [fuse-bridge.c:2342:fuse_statfs_cbk] glusterfs-fuse: 43: ERR => -1 (Transport endpoint is not connected)<br>
<br>
clientlog: (rcnetwork down on gfs-01-02)<br>
[2009-12-18 12:11:48] E [client-protocol.c:415:client_ping_timer_expired] office-data02-rr: Server <a href="http://192.168.11.82:7000" target="_blank">192.168.11.82:7000</a> has not responded in the last 42 seconds, disconnecting.<br>
[2009-12-18 12:11:48] E [saved-frames.c:165:saved_frames_unwind] office-data02-rr: forced unwinding frame type(1) op(STAT)<br>
[2009-12-18 12:11:48] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 58: STAT() /funny => -1 (Transport endpoint is not connected)<br>
[2009-12-18 12:11:48] E [saved-frames.c:165:saved_frames_unwind] office-data02-rr: forced unwinding frame type(1) op(STATFS)<br>
[2009-12-18 12:11:48] W [fuse-bridge.c:2342:fuse_statfs_cbk] glusterfs-fuse: 59: ERR => -1 (Transport endpoint is not connected)<br>
[2009-12-18 12:11:48] E [saved-frames.c:165:saved_frames_unwind] office-data02-rr: forced unwinding frame type(2) op(PING)<br>
[2009-12-18 12:11:48] N [client-protocol.c:6972:notify] office-data02-rr: disconnected<br>
[2009-12-18 12:11:48] N [client-protocol.c:6224:client_setvolume_cbk] office-data02-rr: Connected to <a href="http://192.168.11.11:7000" target="_blank">192.168.11.11:7000</a>, attached to remote volume 'office-data02'.<br>
[2009-12-18 12:11:48] N [client-protocol.c:6224:client_setvolume_cbk] office-data02-rr: Connected to <a href="http://192.168.11.11:7000" target="_blank">192.168.11.11:7000</a>, attached to remote volume 'office-data02'.<br>
<br>
<br>
Did we do anything wrong? Are the volfile wrong? This couldnt be a normal behavior?<br>
<br>
gfs-01-01: /etc/glusterfs/export-office-data02-server_repl_gfs-01-01.vol<br>
<br>
# export-web-data-server_repl<br>
# gfs-01-01 /GFS/web-data<br>
# gfs-01-02 /GFS/web-data<br>
<br>
volume posix<br>
type storage/posix<br>
option directory /GFS/office-data02<br>
end-volume<br>
<br>
volume locks<br>
type features/locks<br>
subvolumes posix<br>
end-volume<br>
<br>
volume posix-remote<br>
type protocol/client<br>
option transport-type tcp<br>
option ping-timeout 5<br>
option remote-host gfs-01-02<br>
option remote-port 7000<br>
option remote-subvolume locks<br>
end-volume<br>
<br>
volume gfs-replicate<br>
type cluster/replicate<br>
subvolumes posix-remote<br>
end-volume<br>
<br>
volume writebehind<br>
type performance/write-behind<br>
option cache-size 2MB<br>
option flush-behind on<br>
subvolumes gfs-replicate<br>
end-volume<br>
<br>
volume office-data02<br>
type performance/io-threads<br>
option thread-count 32 # default is 16<br>
subvolumes writebehind<br>
end-volume<br>
<br>
volume quickread<br>
type performance/quick-read<br>
option cache-timeout 1<br>
option max-file-size 512kB<br>
# subvolumes web-data<br>
subvolumes office-data02<br>
end-volume<br>
<br>
volume server<br>
type protocol/server<br>
option transport-type tcp<br>
option transport.socket.listen-port 7000<br>
option auth.addr.office-data02.allow 192.xxxx.xxx.*<br>
option auth.addr.locks.allow 192.xxxx.xxx.*<br>
subvolumes office-data02 locks<br>
end-volume<br>
<br>
gfs-01-02: /etc/glusterfs/export-office-data02-server_repl_gfs-01-02.vol<br>
# export-office-data02-server_repl<br>
# gfs-01-01 /GFS/office-data02<br>
# gfs-01-02 /GFS/office-data02<br>
<br>
volume posix<br>
type storage/posix<br>
option directory /GFS/office-data02<br>
end-volume<br>
<br>
volume locks<br>
type features/locks<br>
subvolumes posix<br>
end-volume<br>
<br>
volume posix-remote<br>
type protocol/client<br>
option transport-type tcp<br>
option ping-timeout 5<br>
option remote-host gfs-01-01<br>
option remote-port 7000<br>
option remote-subvolume locks<br>
end-volume<br>
<br>
volume gfs-replicate<br>
type cluster/replicate<br>
subvolumes posix-remote<br>
end-volume<br>
<br>
volume writebehind<br>
type performance/write-behind<br>
option cache-size 2MB<br>
option flush-behind on<br>
subvolumes gfs-replicate<br>
end-volume<br>
<br>
volume office-data02<br>
type performance/io-threads<br>
option thread-count 32 # default is 16<br>
subvolumes writebehind<br>
end-volume<br>
<br>
volume quickread<br>
type performance/quick-read<br>
option cache-timeout 1<br>
option max-file-size 512kB<br>
# subvolumes web-data<br>
subvolumes office-data02<br>
end-volume<br>
<br>
volume server<br>
type protocol/server<br>
option transport-type tcp<br>
option transport.socket.listen-port 7000<br>
option auth.addr.office-data02.allow 192.xxx.xxx.*<br>
option auth.addr.locks.allow 192.xxx.xxx.*<br>
subvolumes office-data02 locks<br>
end-volume<br>
<br>
client volfile:<br>
volume office-data02-rr<br>
type protocol/client<br>
option transport-type tcp<br>
option remote-host cluster-gfs.xxxx.xxxxx # RRDNS<br>
option remote-port 7000<br>
option remote-subvolume office-data02<br>
end-volume<br>
<br>
volume readahead<br>
type performance/read-ahead<br>
option page-count 16 # cache per file = (page-count x page-size)<br>
option force-atime-update off # default is off<br>
subvolumes office-data02-rr<br>
end-volume<br>
<br>
volume writebehind<br>
type performance/write-behind<br>
option cache-size 512MB # default is equal to aggregate-size<br>
option flush-behind on # default is 'off'<br>
subvolumes readahead<br>
end-volume<br>
<br>
volume iocache<br>
type performance/io-cache<br>
option cache-size 64MB # default is 32MB<br>
# option priority *.h:3,*.html:2,*:1 # default is '*:0'<br>
option cache-timeout 3 # default is 1 second<br>
subvolumes writebehind<br>
end-volume<br>
<br>
volume quickread<br>
type performance/quick-read<br>
option cache-timeout 1<br>
option max-file-size 512<br>
subvolumes iocache<br>
end-volume<br>
<br>
volume stat-prefetch<br>
type performance/stat-prefetch<br>
subvolumes quickread<br>
end-volume<br>
<br>
<br>
Roland Fischer<br>
<br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div><br><br clear="all"><br>-- <br>Raghavendra G<br><br>