<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap:break-word">
I have been having issues with Gluster for the past couple of weeks on my scratch server (mostly stability). Today, Gluster keeps crashing and will only stay up for a few seconds at a time. The nfs log file contains these messages:
<div><br>
</div>
<div>
<div style="margin:0px; font-size:11px; font-family:Menlo">READDIRP(40)) called at 2014-05-20 14:56:58.933291</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.568960] E [client3_1-fops.c:1937:client3_1_readdirp_cbk] 0-hpcscratch-client-0: remote operation failed: Transport endpoint is not connected</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.568994] E [rpc-clnt.c:341:saved_frames_unwind] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7f193f7b0729] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
[0x7f193f7afeee] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f193f7afe5e]))) 0-hpcscratch-client-0: forced unwinding frame type(GlusterFS 3.1) op(READDIRP(40)) called at 2014-05-20 14:56:58.933348</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569007] E [client3_1-fops.c:1937:client3_1_readdirp_cbk] 0-hpcscratch-client-0: remote operation failed: Transport endpoint is not connected</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569043] E [rpc-clnt.c:341:saved_frames_unwind] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7f193f7b0729] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
[0x7f193f7afeee] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f193f7afe5e]))) 0-hpcscratch-client-0: forced unwinding frame type(GlusterFS 3.1) op(READDIRP(40)) called at 2014-05-20 14:56:58.933405</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569057] E [client3_1-fops.c:1937:client3_1_readdirp_cbk] 0-hpcscratch-client-0: remote operation failed: Transport endpoint is not connected</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569484] E [rpc-clnt.c:341:saved_frames_unwind] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7f193f7b0729] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
[0x7f193f7afeee] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f193f7afe5e]))) 0-hpcscratch-client-0: forced unwinding frame type(GlusterFS 3.1) op(READDIRP(40)) called at 2014-05-20 14:56:58.933484</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569487] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-hpcscratch-client-0: failed to submit rpc-request (XID: 0x289032x Program: GlusterFS 3.1, ProgVers: 310, Proc: 20) to rpc-transport
(hpcscratch-client-0)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569513] E [client3_1-fops.c:1937:client3_1_readdirp_cbk] 0-hpcscratch-client-0: remote operation failed: Transport endpoint is not connected</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569538] E [client3_1-fops.c:2132:client3_1_opendir_cbk] 0-hpcscratch-client-0: remote operation failed: Transport endpoint is not connected</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569570] W [glusterfsd.c:727:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x30e3ce5ccd] (-->/lib64/libpthread.so.0() [0x30e40077f1] (-->/opt/glusterfs/3.2.6/sbin/glusterfs(glusterfs_sigwaiter+0x17c)
[0x40477c]))) 0-: received signum (15), shutting down</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569585] E [rpc-clnt.c:341:saved_frames_unwind] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7f193f7b0729] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
[0x7f193f7afeee] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f193f7afe5e]))) 0-hpcscratch-client-0: forced unwinding frame type(GlusterFS 3.1) op(READDIRP(40)) called at 2014-05-20 14:56:58.933615</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569631] E [client3_1-fops.c:1937:client3_1_readdirp_cbk] 0-hpcscratch-client-0: remote operation failed: Transport endpoint is not connected</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569660] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-hpcscratch-client-0: failed to submit rpc-request (XID: 0x289033x Program: GlusterFS 3.1, ProgVers: 310, Proc: 20) to rpc-transport
(hpcscratch-client-0)</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569677] E [rpc-clnt.c:341:saved_frames_unwind] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7f193f7b0729] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
[0x7f193f7afeee] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f193f7afe5e]))) 0-hpcscratch-client-0: forced unwinding frame type(GlusterFS 3.1) op(READDIRP(40)) called at 2014-05-20 14:56:58.933694</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569707] E [client3_1-fops.c:1937:client3_1_readdirp_cbk] 0-hpcscratch-client-0: remote operation failed: Transport endpoint is not connected</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569688] E [client3_1-fops.c:2132:client3_1_opendir_cbk] 0-hpcscratch-client-0: remote operation failed: Transport endpoint is not connected</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569775] E [rpc-clnt.c:341:saved_frames_unwind] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7f193f7b0729] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
[0x7f193f7afeee] (-->/opt/glusterfs/3.2.6/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f193f7afe5e]))) 0-hpcscratch-client-0: forced unwinding frame type(GlusterFS 3.1) op(READDIRP(40)) called at 2014-05-20 14:56:58.933746</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:51.569795] E [client3_1-fops.c:1937:client3_1_readdirp_cbk] 0-hpcscratch-client-0: remote operation failed: Transport endpoint is not connected</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">[2014-05-20 15:01:52.687996] I [glusterfsd.c:1493:main] 0-/opt/glusterfs/3.2.6/sbin/glusterfs: Started running /opt/glusterfs/3.2.6/sbin/glusterfs version 3.2.6</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><br>
</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">I have also noticed this in the volume log file:</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><br>
</div>
<div style="margin:0px; font-size:11px; font-family:Menlo">
<div style="margin:0px">[2014-05-20 15:01:52.720687] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.50.173.147:1016)</div>
<div style="margin:0px"><br>
</div>
<div style="margin:0px">Does anyone have a clue on what could be causing this or any suggestions on fixing it?</div>
<div style="margin:0px"><br>
</div>
</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><br>
</div>
<div style="margin:0px; font-size:11px; font-family:Menlo"><br>
</div>
<div>
<div style="color:rgb(0,0,0); font-family:Helvetica; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:normal; orphans:2; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; word-wrap:break-word">
<div>
<div>__________________________</div>
<div>Mike Jarsulic</div>
<div>Sr. System Administrator</div>
<div>Center for Research Informatics | University of Chicago</div>
<div>Phone: (773) 702-2066 </div>
<div>Email: <a href="mailto:mjarsulic@bsd.uchicago.edu">mjarsulic@bsd.uchicago.edu</a></div>
</div>
<div><br>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br>
</div>
<br>
<hr size="1">
<font color="gray" face="Verdana, Arial" size="1">This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended
recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.<br>
<br>
Thank you. </font>
</body>
</html>