<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=gb2312"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:ËÎÌå;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@ËÎÌå";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
/* Page Definitions */
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=ZH-CN link=blue vlink=purple style='text-justify-trim:punctuation'><div class=WordSection1><p class=MsoNormal><span lang=EN-US>Hi, everyone:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>We deploy GlusterFS3.2.7 with 24 servers and each server has 12 disks using for brick server.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>The volume type is DHT + AFR(replica=3) and transport type is socket.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Use native client and each server has 8 mount points.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The gluster client will log some error once in a while, following is an example.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>We google these error, but not find any method to resolve it. Can give me some help, thanks!<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:red'>[2013-06-14 15:58:41.913210] W [client-handshake.c:264:client_ping_cbk] 0-gfs1-client-174: timer must have expired<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:red'>[2013-06-14 15:58:41.913439] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x32f320e6e8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x32f320de7d] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x32f320ddde]))) 0-gfs1-client-184: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-06-14 15:57:36.516579<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 15:58:41.913456] W [client-handshake.c:264:client_ping_cbk] 0-gfs1-client-184: timer must have expired<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 15:58:41.913526] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x32f320e6e8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x32f320de7d] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x32f320ddde]))) 0-gfs1-client-187: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-06-14 15:57:36.516591<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 15:58:41.913542] W [client-handshake.c:264:client_ping_cbk] 0-gfs1-client-187: timer must have expired<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 15:58:41.913662] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x32f320e6e8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x32f320de7d] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x32f320ddde]))) 0-gfs1-client-170: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-06-14 15:57:36.516554<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 15:58:41.913678] W [client-handshake.c:264:client_ping_cbk] 0-gfs1-client-170: timer must have expired<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 15:58:41.914089] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x32f320e6e8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x32f320de7d] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x32f320ddde]))) 0-gfs1-client-180: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-06-14 15:57:36.516543<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 15:58:41.914115] W [client-handshake.c:264:client_ping_cbk] 0-gfs1-client-180: timer must have expired<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:red'>[2013-06-14 16:16:36.467651] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-51: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11081156x sent = 2013-06-14 15:46:27.573575. timeout = 1800<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 16:16:36.467723] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-51: remote operation failed: Transport endpoint is not connected<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 16:32:07.91967] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-48: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11115092x sent = 2013-06-14 16:02:02.712824. timeout = 1800<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 16:32:07.92014] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-48: remote operation failed: Transport endpoint is not connected<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 16:35:42.221471] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-48: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11115696x sent = 2013-06-14 16:05:41.199990. timeout = 1800<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 16:35:42.221586] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-48: remote operation failed: Transport endpoint is not connected<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 16:36:12.766153] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-48: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11115817x sent = 2013-06-14 16:06:02.592259. timeout = 1800<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 16:36:12.766201] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-48: remote operation failed: Transport endpoint is not connected<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 16:47:32.203692] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-51: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11095024x sent = 2013-06-14 16:17:31.966654. timeout = 1800<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 16:47:32.203766] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-51: remote operation failed: Transport endpoint is not connected<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 17:17:42.983248] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-51: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11112188x sent = 2013-06-14 16:47:32.210391. timeout = 1800<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 17:17:42.983337] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-51: remote operation failed: Transport endpoint is not connected<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 17:17:42.983413] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-51: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11112186x sent = 2013-06-14 16:47:32.206834. timeout = 1800<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 17:17:42.983449] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-51: remote operation failed: Transport endpoint is not connected<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[2013-06-14 17:19:15.201853] W [dict.c:418:dict_unref] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x32f320e4e5] (-->/usr/lib/glusterfs/3.2.7/xlator/protocol/client.so(client3_1_fstat_cbk+0x343) [0x7fd4d4c4a373] (-->/usr/lib/glusterfs/3.2.7/xlator/cluster/replicate.so(afr_sh_data_fstat_cbk+0x1eb) [0x7fd4d49ed90b]))) 0-dict: dict is NULL<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Volume info as below:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>[root@bj-nx-cip-w83 ~]# gluster volume info<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Volume Name: gfs1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Type: Distributed-Replicate<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Status: Started<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Number of Bricks: 94 x 3 = 282<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Transport-type: tcp<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Bricks:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Brick1: 10.0.11.81:/xmail/disk1/gfs1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Brick2: 10.0.12.71:/xmail/disk1/gfs1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Brick3: 10.0.13.91:/xmail/disk1/gfs1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Brick4: 10.0.11.82:/xmail/disk1/gfs1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Brick5: 10.0.12.72:/xmail/disk1/gfs1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Brick6: 10.0.13.92:/xmail/disk1/gfs1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>¡¡<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Brick280: 10.0.13.88:/xmail/disk12/gfs1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Brick281: 10.0.11.78:/xmail/disk12/gfs1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Brick282: 10.0.12.68:/xmail/disk12/gfs1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Options Reconfigured:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>performance.read-ahead: on<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>performance.io-thread-count: 16<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>diagnostics.client-log-level: WARNING<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>diagnostics.brick-log-level: WARNING<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>nfs.disable: on<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>performance.io-cache: off<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>performance.write-behind: off<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>--------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Cailiang song<o:p></o:p></span></p></div></body></html>