[2012-06-08 10:21:56.123954] I [glusterfsd.c:1493:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.2.5 [2012-06-08 10:21:56.170531] W [write-behind.c:3023:init] 0-distribute-write-behind: disabling write-behind for first 0 bytes [2012-06-08 10:21:56.173178] I [client.c:1935:notify] 0-distribute-client-0: parent translators are ready, attempting connect on transport [2012-06-08 10:21:56.176678] I [client.c:1935:notify] 0-distribute-client-1: parent translators are ready, attempting connect on transport Given volfile: +------------------------------------------------------------------------------+ 1: volume distribute-client-0 2: type protocol/client 3: option remote-host 172.17.13.31 4: option remote-subvolume /export/data/watch 5: option transport-type tcp 6: end-volume 7: 8: volume distribute-client-1 9: type protocol/client 10: option remote-host 172.17.13.32 11: option remote-subvolume /export/data/watch 12: option transport-type tcp 13: end-volume 14: 15: volume distribute-dht 16: type cluster/distribute 17: subvolumes distribute-client-0 distribute-client-1 18: end-volume 19: 20: volume distribute-write-behind 21: type performance/write-behind 22: subvolumes distribute-dht 23: end-volume 24: 25: volume distribute-read-ahead 26: type performance/read-ahead 27: subvolumes distribute-write-behind 28: end-volume 29: 30: volume distribute-io-cache 31: type performance/io-cache 32: subvolumes distribute-read-ahead 33: end-volume 34: 35: volume distribute-quick-read 36: type performance/quick-read 37: subvolumes distribute-io-cache 38: end-volume 39: 40: volume distribute-stat-prefetch 41: type performance/stat-prefetch 42: subvolumes distribute-quick-read 43: end-volume 44: 45: volume distribute 46: type debug/io-stats 47: option latency-measurement off 48: option count-fop-hits off 49: subvolumes distribute-stat-prefetch 50: end-volume +------------------------------------------------------------------------------+ [2012-06-08 10:21:56.180584] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-distribute-client-0: changing port to 24009 (from 0) [2012-06-08 10:21:56.180685] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-distribute-client-1: changing port to 24009 (from 0) [2012-06-08 10:22:00.175056] I [client-handshake.c:1090:select_server_supported_programs] 0-distribute-client-0: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2012-06-08 10:22:00.175363] I [client-handshake.c:913:client_setvolume_cbk] 0-distribute-client-0: Connected to 172.17.13.31:24009, attached to remote volume '/export/data/watch'. [2012-06-08 10:22:00.178525] I [client-handshake.c:1090:select_server_supported_programs] 0-distribute-client-1: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2012-06-08 10:22:00.178773] I [client-handshake.c:913:client_setvolume_cbk] 0-distribute-client-1: Connected to 172.17.13.32:24009, attached to remote volume '/export/data/watch'. [2012-06-08 10:22:00.181923] I [fuse-bridge.c:3339:fuse_graph_setup] 0-fuse: switched to graph 0 [2012-06-08 10:22:00.182130] I [fuse-bridge.c:2927:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10 [2012-08-19 01:19:43.738071] W [socket.c:204:__socket_rwv] 0-glusterfs: readv failed (Connection timed out) [2012-08-19 01:19:43.746930] W [socket.c:1494:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Connection timed out), peer (172.17.13.32:24007) [2012-08-19 01:20:00.482357] E [socket.c:1685:socket_connect_finish] 0-glusterfs: connection to 172.17.13.32:24007 failed (No route to host) [2012-08-19 01:20:02.758554] W [socket.c:204:__socket_rwv] 0-distribute-client-1: readv failed (Connection reset by peer) [2012-08-19 01:20:02.758594] W [socket.c:1494:__socket_proto_state_machine] 0-distribute-client-1: reading from socket failed. Error (Connection reset by peer), peer (172.17.13.32:24009) [2012-08-19 01:20:02.770028] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.2085 [2012-08-19 01:20:02.775388] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.775459] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.2407 [2012-08-19 01:20:02.775478] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.775521] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.2696 [2012-08-19 01:20:02.775537] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.775576] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.2820 [2012-08-19 01:20:02.775592] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.775631] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.3140 [2012-08-19 01:20:02.775652] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.775692] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.3693 [2012-08-19 01:20:02.775708] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.775747] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.4050 [2012-08-19 01:20:02.775762] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.775801] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.4364 [2012-08-19 01:20:02.775826] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.775868] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.4643 [2012-08-19 01:20:02.775883] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.775924] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.4818 [2012-08-19 01:20:02.775939] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.775977] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.5077 [2012-08-19 01:20:02.775992] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776032] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.5303 [2012-08-19 01:20:02.776047] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776086] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.6212 [2012-08-19 01:20:02.776101] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776140] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.6528 [2012-08-19 01:20:02.776155] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776194] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.6698 [2012-08-19 01:20:02.776218] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776259] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.7024 [2012-08-19 01:20:02.776276] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776326] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.7220 [2012-08-19 01:20:02.776342] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776382] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.7530 [2012-08-19 01:20:02.776397] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776436] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.7713 [2012-08-19 01:20:02.776451] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776490] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.7835 [2012-08-19 01:20:02.776505] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776544] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.8243 [2012-08-19 01:20:02.776559] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776599] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.8366 [2012-08-19 01:20:02.776622] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776669] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.8850 [2012-08-19 01:20:02.776684] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776724] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.9548 [2012-08-19 01:20:02.776739] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776779] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.9644 [2012-08-19 01:20:02.776794] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776833] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.9848 [2012-08-19 01:20:02.776848] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776888] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:23.10220 [2012-08-19 01:20:02.776903] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776942] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.2476 [2012-08-19 01:20:02.776957] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.776995] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.3169 [2012-08-19 01:20:02.777018] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777059] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.3274 [2012-08-19 01:20:02.777074] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777113] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.3381 [2012-08-19 01:20:02.777128] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777166] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.3480 [2012-08-19 01:20:02.777181] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777220] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.3581 [2012-08-19 01:20:02.777235] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777275] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.4003 [2012-08-19 01:20:02.777300] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777341] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.4152 [2012-08-19 01:20:02.777357] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777396] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.4298 [2012-08-19 01:20:02.777420] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777460] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.4421 [2012-08-19 01:20:02.777476] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777515] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.4511 [2012-08-19 01:20:02.777530] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777568] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.4933 [2012-08-19 01:20:02.777582] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777621] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.5455 [2012-08-19 01:20:02.777641] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777681] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.5562 [2012-08-19 01:20:02.777696] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777734] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.5664 [2012-08-19 01:20:02.777749] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777787] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.5752 [2012-08-19 01:20:02.777812] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777852] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.6146 [2012-08-19 01:20:02.777867] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777906] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.6334 [2012-08-19 01:20:02.777921] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.777959] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.6435 [2012-08-19 01:20:02.777974] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778013] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.6615 [2012-08-19 01:20:02.778028] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778066] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.6980 [2012-08-19 01:20:02.778081] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778119] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:24.7069 [2012-08-19 01:20:02.778134] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778173] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:25.2217 [2012-08-19 01:20:02.778187] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778234] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:25.2317 [2012-08-19 01:20:02.778250] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778295] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:25.2491 [2012-08-19 01:20:02.778313] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778353] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:25.3139 [2012-08-19 01:20:02.778368] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778407] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:25.3226 [2012-08-19 01:20:02.778421] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778460] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:25.3452 [2012-08-19 01:20:02.778474] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778514] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:26.1015 [2012-08-19 01:20:02.778529] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778567] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:26.2095 [2012-08-19 01:20:02.778581] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778629] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:26.2343 [2012-08-19 01:20:02.778651] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778690] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:26.2798 [2012-08-19 01:20:02.778705] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778744] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:26.3629 [2012-08-19 01:20:02.778759] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778797] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:26.4090 [2012-08-19 01:20:02.778812] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778850] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:26.4199 [2012-08-19 01:20:02.778865] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778904] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:26.4293 [2012-08-19 01:20:02.778918] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.778957] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:26.4824 [2012-08-19 01:20:02.778971] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.779020] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:26.4930 [2012-08-19 01:20:02.779036] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.779074] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:26.5575 [2012-08-19 01:20:02.779089] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.779128] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:26.5705 [2012-08-19 01:20:02.779143] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.779182] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:27.2176 [2012-08-19 01:20:02.779196] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.779235] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:27.2759 [2012-08-19 01:20:02.779250] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.779295] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:27.2926 [2012-08-19 01:20:02.779312] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.779352] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:27.3117 [2012-08-19 01:20:02.779367] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.779406] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:27.3266 [2012-08-19 01:20:02.779430] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.779470] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:27.3417 [2012-08-19 01:20:02.779485] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.779524] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:19:27.3503 [2012-08-19 01:20:02.779538] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-1: remote operation failed: Transport endpoint is not connected [2012-08-19 01:20:02.779577] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-1: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2012-08-19 01:19:53.461232 [2012-08-19 01:20:02.780891] W [client-handshake.c:258:client_ping_cbk] 0-distribute-client-1: socket or ib related error [2012-08-19 01:20:02.781043] I [client.c:1883:client_rpc_notify] 0-distribute-client-1: disconnected [2012-08-19 01:20:03.506197] I [glusterfsd-mgmt.c:637:mgmt_getspec_cbk] 0-: No change in volfile, continuing [2012-08-19 01:20:03.538433] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216532: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:03.539249] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-5924943517): failed to get fd ctx. EBADFD [2012-08-19 01:20:03.539274] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:03.661875] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216535: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:03.662457] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-4893168229): failed to get fd ctx. EBADFD [2012-08-19 01:20:03.662482] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:04.3218] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216538: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:04.3399] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (759076083): failed to get fd ctx. EBADFD [2012-08-19 01:20:04.3430] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:04.3507] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216540: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:04.3706] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-4009986849): failed to get fd ctx. EBADFD [2012-08-19 01:20:04.3736] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:04.3860] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216542: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:04.4093] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (2126345417): failed to get fd ctx. EBADFD [2012-08-19 01:20:04.4115] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:04.4212] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-3673927339): failed to get fd ctx. EBADFD [2012-08-19 01:20:04.4241] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:04.448802] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216550: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:04.449574] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-5322905079): failed to get fd ctx. EBADFD [2012-08-19 01:20:04.449602] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:05.3202] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-4084726743): failed to get fd ctx. EBADFD [2012-08-19 01:20:05.3242] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:05.3532] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-7245322353): failed to get fd ctx. EBADFD [2012-08-19 01:20:05.3560] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:06.3767] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-6663609839): failed to get fd ctx. EBADFD [2012-08-19 01:20:06.3799] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:06.3877] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216558: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:06.4072] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216560: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:06.4479] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-2027313707): failed to get fd ctx. EBADFD [2012-08-19 01:20:06.4502] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:06.4599] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-6604019939): failed to get fd ctx. EBADFD [2012-08-19 01:20:06.4619] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:06.4865] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-6062391659): failed to get fd ctx. EBADFD [2012-08-19 01:20:06.4886] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:06.407650] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216570: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:06.408251] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-2107088771): failed to get fd ctx. EBADFD [2012-08-19 01:20:06.408281] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:07.3017] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-885653357): failed to get fd ctx. EBADFD [2012-08-19 01:20:07.3047] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:07.3144] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-31236935): failed to get fd ctx. EBADFD [2012-08-19 01:20:07.3164] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:07.3457] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-7007166575): failed to get fd ctx. EBADFD [2012-08-19 01:20:07.3484] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:07.73772] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216581: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:07.74074] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-5038328345): failed to get fd ctx. EBADFD [2012-08-19 01:20:07.74098] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:08.3971] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-162743769): failed to get fd ctx. EBADFD [2012-08-19 01:20:08.4010] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:08.4110] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216589: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:08.4432] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216592: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:08.4700] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-2265974431): failed to get fd ctx. EBADFD [2012-08-19 01:20:08.4724] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:08.5224] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216599: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:08.5446] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216601: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:08.5525] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216602: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:08.5674] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-7079812767): failed to get fd ctx. EBADFD [2012-08-19 01:20:08.5696] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:08.5769] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216604: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:08.5846] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216605: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:08.5942] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-5449134049): failed to get fd ctx. EBADFD [2012-08-19 01:20:08.5961] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:08.6037] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216607: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:08.6110] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-2478468081): failed to get fd ctx. EBADFD [2012-08-19 01:20:08.6129] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:08.6198] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-3930011207): failed to get fd ctx. EBADFD [2012-08-19 01:20:08.6217] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:08.6285] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-5979072583): failed to get fd ctx. EBADFD [2012-08-19 01:20:08.6303] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:08.6402] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-4611804387): failed to get fd ctx. EBADFD [2012-08-19 01:20:08.6439] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:08.6529] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-8245780967): failed to get fd ctx. EBADFD [2012-08-19 01:20:08.6548] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:08.6649] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-7313071749): failed to get fd ctx. EBADFD [2012-08-19 01:20:08.6669] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:08.958007] W [client3_1-fops.c:3971:client3_1_statfs] 0-distribute-client-1: failed to send the fop: Transport endpoint is not connected [2012-08-19 01:20:09.5159] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-4471888319): failed to get fd ctx. EBADFD [2012-08-19 01:20:09.5193] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:09.5318] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-4867961701): failed to get fd ctx. EBADFD [2012-08-19 01:20:09.5344] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:09.5751] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216632: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:09.5860] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-914872913): failed to get fd ctx. EBADFD [2012-08-19 01:20:09.5882] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:09.5975] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-2067014735): failed to get fd ctx. EBADFD [2012-08-19 01:20:09.5995] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:09.6283] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216637: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:09.6363] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216638: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:09.6576] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-3292285267): failed to get fd ctx. EBADFD [2012-08-19 01:20:09.6599] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:09.6871] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216643: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:09.6950] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-914872913): failed to get fd ctx. EBADFD [2012-08-19 01:20:09.6970] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:09.6986] W [fuse-bridge.c:850:fuse_err_cbk] 0-glusterfs-fuse: 228216644: FLUSH() ERR => -1 (File descriptor in bad state) [2012-08-19 01:20:09.7101] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-374841961): failed to get fd ctx. EBADFD [2012-08-19 01:20:09.7121] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:09.7188] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-1170706087): failed to get fd ctx. EBADFD [2012-08-19 01:20:09.7207] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:09.7276] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-2449797547): failed to get fd ctx. EBADFD [2012-08-19 01:20:09.7294] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:10.1613] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216651: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:10.1711] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216652: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:10.1805] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (863015789): failed to get fd ctx. EBADFD [2012-08-19 01:20:10.1825] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:10.2999] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216656: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:10.71718] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216658: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:10.71962] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-1074258427): failed to get fd ctx. EBADFD [2012-08-19 01:20:10.71986] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:11.1646] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216661: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:11.1908] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (882941469): failed to get fd ctx. EBADFD [2012-08-19 01:20:11.1932] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:11.3194] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216666: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:11.636045] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216671: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:11.636346] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-1013618619): failed to get fd ctx. EBADFD [2012-08-19 01:20:11.636368] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:12.3342] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-4612707937): failed to get fd ctx. EBADFD [2012-08-19 01:20:12.3371] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:12.3493] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216678: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:13.3556] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216731: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:13.3820] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216733: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:13.4024] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216735: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:13.4361] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216738: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:13.4449] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216739: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:13.4782] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-1518758995): failed to get fd ctx. EBADFD [2012-08-19 01:20:13.4807] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:13.5362] W [client3_1-fops.c:3604:client3_1_writev] 0-distribute-client-1: (-3622414097): failed to get fd ctx. EBADFD [2012-08-19 01:20:13.5385] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:13.5463] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-4439728235): failed to get fd ctx. EBADFD [2012-08-19 01:20:13.5484] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:13.5576] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-3636441575): failed to get fd ctx. EBADFD [2012-08-19 01:20:13.5596] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:13.5688] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-5948227409): failed to get fd ctx. EBADFD [2012-08-19 01:20:13.5715] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:13.5821] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-5067029669): failed to get fd ctx. EBADFD [2012-08-19 01:20:13.5842] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:13.5934] W [client3_1-fops.c:3662:client3_1_flush] 0-distribute-client-1: (-3622414097): failed to get fd ctx. EBADFD [2012-08-19 01:20:13.5962] W [client3_1-fops.c:3692:client3_1_flush] 0-distribute-client-1: failed to send the fop: File descriptor in bad state [2012-08-19 01:20:13.5980] W [fuse-bridge.c:850:fuse_err_cbk] 0-glusterfs-fuse: 228216754: FLUSH() ERR => -1 (File descriptor in bad state) [2012-08-19 01:20:13.513415] I [client-handshake.c:1090:select_server_supported_programs] 0-distribute-client-1: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2012-08-19 01:20:13.513773] I [client-handshake.c:913:client_setvolume_cbk] 0-distribute-client-1: Connected to 172.17.13.32:24009, attached to remote volume '/export/data/watch'. [2012-08-19 01:20:13.513927] I [client-handshake.c:779:client_post_handshake] 0-distribute-client-1: 896 fds open - Delaying child_up until they are re-opened [2012-08-19 01:20:13.527867] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.527925] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.544737] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.546658] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.547666] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.547721] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.549073] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.573107] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.580158] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.591979] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.595901] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.663125] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.734223] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.771281] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.810391] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.830208] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.832752] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.833326] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.834967] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.835173] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.835559] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.835736] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.835830] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.835958] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.844562] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.845668] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.848550] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-distribute-client-1: sending release on fd [2012-08-19 01:20:13.850020] I [client-lk.c:617:decrement_reopen_fd_count] 0-distribute-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP [2012-08-19 01:20:14.3382] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216762: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:14.3662] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216764: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:14.3877] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216766: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:14.4300] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216771: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:14.4403] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216772: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:14.5056] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216778: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:15.3403] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216788: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:16.4669] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216814: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:16.6007] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216826: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:17.3101] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216836: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:17.3351] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216838: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:17.4226] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216846: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:17.4406] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216848: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:17.4977] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216854: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:17.406003] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216862: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:18.3264] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216869: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:18.3377] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216870: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:18.3618] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216872: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:18.4421] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216879: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:18.5138] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216886: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:18.5730] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216892: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:18.6278] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216898: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:18.6518] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216901: WRITE => -1 (File descriptor in bad state) [2012-08-19 01:20:19.4874] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216924: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:19.4997] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216925: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:19.5594] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216930: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:19.5948] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216933: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:19.6625] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228216939: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:22.762067] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228217013: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:23.4343] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228217031: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:23.4922] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228217037: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:24.2054] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228217043: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:24.2405] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228217045: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:24.2493] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228217046: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:24.3463] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228217049: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:24.3550] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228217050: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:24.3641] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228217051: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:20:30.3473] W [fuse-bridge.c:1828:fuse_writev_cbk] 0-glusterfs-fuse: 228217175: WRITE => -1 (Transport endpoint is not connected) [2012-08-19 01:21:28.588776] C [client-handshake.c:121:rpc_client_ping_timer_expired] 0-distribute-client-0: server 172.17.13.31:24009 has not responded in the last 42 seconds, disconnecting. [2012-08-19 01:21:28.589033] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-0: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:20:04.4021 [2012-08-19 01:21:28.589078] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-0: remote operation failed: Transport endpoint is not connected [2012-08-19 01:21:28.589135] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-0: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2012-08-19 01:20:04.4032 [2012-08-19 01:21:28.589153] W [client-handshake.c:264:client_ping_cbk] 0-distribute-client-0: timer must have expired [2012-08-19 01:21:28.589190] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b68c117b659] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x2b68c117ae1e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x2b68c117ad8e]))) 0-distribute-client-0: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-08-19 01:20:04.435396 [2012-08-19 01:21:28.589227] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-0: remote operation failed: Transport endpoint is not connected [2012-08-19 01:21:28.592930] I [socket.c:2275:socket_submit_request] 0-distribute-client-0: not connected (priv->connected = 0) [2012-08-19 01:21:28.592955] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-distribute-client-0: failed to submit rpc-request (XID: 0x143670511x Program: GlusterFS 3.1, ProgVers: 310, Proc: 13) to rpc-transport (distribute-client-0) [2012-08-19 01:21:28.592975] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-distribute-client-0: remote operation failed: Transport endpoint is not connected [2012-08-19 01:21:28.592994] W [client3_1-fops.c:3622:client3_1_writev] 0-distribute-client-0: failed to send the fop: Stale NFS file handle pending frames: frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(WRITE) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(LOOKUP) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(LOOKUP) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(STATFS) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) frame : type(1) op(WRITE) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2012-08-19 01:21:28 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.2.5 /lib64/libc.so.6[0x2b68c19e72d0] /usr/local/lib/glusterfs/3.2.5/xlator/performance/write-behind.so(wb_sync_cbk+0x3e)[0x2aaaab609eee] /usr/local/lib/glusterfs/3.2.5/xlator/cluster/distribute.so(dht_writev_cbk+0xcf)[0x2aaaab3df44f] /usr/local/lib/glusterfs/3.2.5/xlator/protocol/client.so(client3_1_writev+0x26d)[0x2aaaab1b015d] /usr/local/lib/glusterfs/3.2.5/xlator/protocol/client.so(client_writev+0xbb)[0x2aaaab1a164b] /usr/local/lib/glusterfs/3.2.5/xlator/cluster/distribute.so(dht_writev+0x180)[0x2aaaab3e4690] /usr/local/lib/glusterfs/3.2.5/xlator/performance/write-behind.so(wb_sync+0x4d6)[0x2aaaab604396] /usr/local/lib/glusterfs/3.2.5/xlator/performance/write-behind.so(wb_do_ops+0x54)[0x2aaaab6046f4] /usr/local/lib/glusterfs/3.2.5/xlator/performance/write-behind.so(wb_process_queue+0xe9)[0x2aaaab6049f9] /usr/local/lib/glusterfs/3.2.5/xlator/performance/write-behind.so(wb_sync_cbk+0xe8)[0x2aaaab609f98] /usr/local/lib/glusterfs/3.2.5/xlator/cluster/distribute.so(dht_writev_cbk+0xcf)[0x2aaaab3df44f] /usr/local/lib/glusterfs/3.2.5/xlator/protocol/client.so(client3_1_writev_cbk+0x4f7)[0x2aaaab1bb227] /usr/local/lib/libgfrpc.so.0(saved_frames_unwind+0x1c2)[0x2b68c117ad12] /usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b68c117ad8e] /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x2b68c117ae1e] /usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9)[0x2b68c117b659] /usr/local/lib/libgfrpc.so.0(rpc_transport_notify+0x27)[0x2b68c11762a7] /usr/local/lib/glusterfs/3.2.5/rpc-transport/socket.so(socket_event_poll_err+0x54)[0x2aaaaad6b5a4] /usr/local/lib/glusterfs/3.2.5/rpc-transport/socket.so(socket_event_handler+0xd8)[0x2aaaaad70fc8] /usr/local/lib/libglusterfs.so.0[0x2b68c0f3a9a1] /usr/local/sbin/glusterfs(main+0x42f)[0x4052ef] /lib64/libc.so.6(__libc_start_main+0xf4)[0x2b68c19d4994] /usr/local/sbin/glusterfs[0x403649] ---------