Hi everyone,<div>I noticed today that one of my brick crashed a week ago. I'll noticed that there was very low load on that particular node.</div><div>My setup: Gluster 3.3.0-1 on Ubuntu 12.04.</div><div><br></div><div>
<div>gluster> volume info</div><div> </div><div>Volume Name: vol0</div><div>Type: Distributed-Replicate</div><div>Volume ID: 211c824d-b71f-4ce2-b56a-98d0ef68cd1a</div><div>Status: Started</div><div>Number of Bricks: 2 x 2 = 4</div>
<div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: unic-prd-os-compute1:/data/brick0</div><div>Brick2: unic-prd-os-compute2:/data/brick0</div><div>Brick3: unic-prd-os-compute3:/data/brick0</div><div>Brick4: unic-prd-os-compute4:/data/brick0</div>
<div>Options Reconfigured:</div><div>performance.cache-size: 256MB</div></div><div><br></div><div>The log shows some strange timeout errors.<br><div><br></div><div><div>[2012-07-24 01:19:05.566729] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-vol0-client-3: server <a href="http://127.0.0.1:24009">127.0.0.1:24009</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2012-07-24 01:19:05.628084] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0xd0) [0x7f600e8d35b0] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f600e8d3220] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f600e8d314e]))) 0-vol0-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-07-24 01:18:19.905575 (xid=0x44083326x)</div>
<div>[2012-07-24 01:19:05.628132] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-vol0-client-3: remote operation failed: Transport endpoint is not connected</div><div>[2012-07-24 01:19:05.628214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0xd0) [0x7f600e8d35b0] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f600e8d3220] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f600e8d314e]))) 0-vol0-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-07-24 01:18:21.717671 (xid=0x44083327x)</div>
<div>[2012-07-24 01:19:05.628226] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-vol0-client-3: remote operation failed: Transport endpoint is not connected</div><div>[2012-07-24 01:19:05.628261] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0xd0) [0x7f600e8d35b0] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f600e8d3220] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f600e8d314e]))) 0-vol0-client-3: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2012-07-24 01:18:23.564658 (xid=0x44083328x)</div>
<div>[2012-07-24 01:19:05.628280] W [client-handshake.c:275:client_ping_cbk] 0-vol0-client-3: timer must have expired</div><div>[2012-07-24 01:19:05.628289] I [client.c:2090:client_rpc_notify] 0-vol0-client-3: disconnected</div>
<div>[2012-07-24 01:19:05.628976] W [client3_1-fops.c:5267:client3_1_finodelk] 0-vol0-client-3: (8d09515c-ca0b-4048-8ebc-604f7c0d0469) remote_fd is -1. EBADFD</div><div>[2012-07-24 01:19:05.629054] W [client3_1-fops.c:5267:client3_1_finodelk] 0-vol0-client-3: (770fdb61-5737-4319-9233-954b3a10dec9) remote_fd is -1. EBADFD</div>
<div>[2012-07-24 01:49:26.750856] E [rpc-clnt.c:208:call_bail] 0-vol0-client-3: bailing out frame type(GF-DUMP) op(DUMP(1)) xid = 0x44083329x sent = 2012-07-24 01:19:16.587767. timeout = 1800</div><div>[2012-07-24 01:49:26.750907] W [client-handshake.c:1819:client_dump_version_cbk] 0-vol0-client-3: received RPC status error</div>
</div></div><div><br></div><div>The gluster console show that the node is online:</div><div><br></div><div><div>gluster> volume status all</div><div>Status of volume: vol0</div><div>Gluster process Port Online Pid</div>
<div>------------------------------------------------------------------------------</div><div>Brick unic-prd-os-compute1:/data/brick0 24009 Y 2166</div><div>Brick unic-prd-os-compute2:/data/brick0 24009 Y 16270</div>
<div>Brick unic-prd-os-compute3:/data/brick0 24009 Y 23231</div><div>Brick unic-prd-os-compute4:/data/brick0 24009 Y 10519</div></div><div><br></div><div>Process is there:</div>
<div><br></div><div><div>root@unic-prd-os-compute4:~# ps aux | grep 10519</div><div>root 10519 34.9 0.0 1433352 22916 ? Rsl Jul07 12063:18 /usr/sbin/glusterfsd -s localhost --volfile-id vol0.unic-prd-os-compute4.data-brick0 -p /var/lib/glusterd/vols/vol0/run/unic-prd-os-compute4-data-brick0.pid -S /tmp/cbafb6c90608cd50a23f2a8c8a4c5da5.socket --brick-name /data/brick0 -l /var/log/glusterfs/bricks/data-brick0.log --xlator-option *-posix.glusterd-uuid=8418fd20-2e16-4033-9341-1f2456ca511d --brick-port 24009 --xlator-option vol0-server.listen-port=24009</div>
</div><div><br></div><div>But it's not connected:</div><div><br></div><div><div>Brick unic-prd-os-compute4:/data/brick0</div><div>Number of entries: 0</div><div>Status: Brick is Not connected</div></div><div><br></div>
<div>The question is now:</div><div>a) How can I detect such outages (with Nagios for example)? As I said, I just noticed it because the load dropped (graph in Zabbix).</div><div>b) Can I just restart glusterd on that node to trigger the self healing?</div>
<div><br></div><div>Cheers,</div><div>Christian</div><div><br></div>