<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Changliang,<br>
<br>
Could you attach the logs of the servers (bricks)?
<blockquote
cite="mid:CAPnto6xO3qsHmD0bOunGAT_dJQi_M9VbNciKBR=RpsAdG6gGOw@mail.gmail.com"
type="cite">
<div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div class="h5">
<blockquote type="cite">
<div> Because to keep availability,we haven't
strace the process.After shudowning the damon,the
cluster recover.</div>
<div> In our case,</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
Pranith was asking for a core dump file or the backtrace (of the
crashed process), not strace output.<br>
<br>
thanks,<br>
krish<br>
<blockquote
cite="mid:CAPnto6xO3qsHmD0bOunGAT_dJQi_M9VbNciKBR=RpsAdG6gGOw@mail.gmail.com"
type="cite">
<div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div class="h5">
<blockquote type="cite">
<div> 10.1.1.64(dfs-client-6): online node,when
the other node(65) restart,cpu usr usage reach
100% (glusterfsd process)</div>
<div> 10.1.1.65(dfs-client-7): offline node,when
it restart,the client nfs mount point
unavailable.</div>
<div> </div>
<div>The nfs.log show that the reason of issue will
be cause by client-6 high cpu usage,there are lots
of error like:</div>
<div><br>
</div>
<div>
<div>[2011-12-14 13:25:53.30308] E
[rpc-clnt.c:197:call_bail] 0-19loudfs-client-6:
bailing out frame type(GlusterFS 3.1)
op(XATTROP(33)) xid = 0x89279937x sent =
2011-12-14 13:25:20.</div>
<div>346007. timeout = 30</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div> <br>
<br>
<div class="gmail_quote">On Wed, Dec 14, 2011 at
6:49 PM, Pranith Kumar K <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:pranithk@gluster.com"
target="_blank">pranithk@gluster.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:
0pt 0pt 0pt 0.8ex; border-left: 1px solid
rgb(204, 204, 204); padding-left: 1ex;">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div> On 12/14/2011 03:06 PM, Changliang
Chen wrote: </div>
</div>
<blockquote type="cite">
<div>
<div> <span style="color: rgb(68, 68,
68); font-family: Ubuntu,sans-serif;
font-size: 13px; line-height: 16px;
background-color: rgb(255, 255,
255);">Hi,we have use glusterfs for
two years. After upgraded to
3.2.5,we discover that when one of
replicate node reboot and startup
the glusterd daemon,the gluster will
crash cause by the other </span>
<div> <span style="color: rgb(68, 68,
68); font-family:
Ubuntu,sans-serif; font-size:
13px; line-height: 16px;
background-color: rgb(255, 255,
255);"><br>
</span></div>
<div><span style="color: rgb(68, 68,
68); font-family:
Ubuntu,sans-serif; font-size:
13px; line-height: 16px;
background-color: rgb(255, 255,
255);">replicate node cpu
usage reach 100%.</span><br
style="color: rgb(68, 68, 68);
font-family: Ubuntu,sans-serif;
font-size: 13px; line-height:
16px; text-align: left;
background-color: rgb(255, 255,
255);">
<br style="color: rgb(68, 68, 68);
font-family: Ubuntu,sans-serif;
font-size: 13px; line-height:
16px; text-align: left;
background-color: rgb(255, 255,
255);">
<span style="color: rgb(68, 68, 68);
font-family: Ubuntu,sans-serif;
font-size: 13px; line-height:
16px; text-align: left;
background-color: rgb(255, 255,
255);">Our gluster info:</span></div>
<div><br style="color: rgb(68, 68,
68); font-family:
Ubuntu,sans-serif; font-size:
13px; line-height: 16px;
text-align: left;
background-color: rgb(255, 255,
255);">
<div style="margin: 0px; padding:
0px; border-width: 0px;
outline-width: 0px;
vertical-align: baseline;
background-color: rgb(255, 255,
255); text-align: left;">
<font face="Ubuntu, sans-serif"
color="#444444"><span
style="line-height: 16px;">Type:
Distributed-Replicate</span></font>
<div style="margin: 0px; padding:
0px; border-width: 0px;
outline-width: 0px;
vertical-align: baseline;
background-color: transparent;">
<font face="Ubuntu, sans-serif"
color="#444444"><span
style="line-height: 16px;">Status:
Started</span></font>
<div style="margin: 0px;
padding: 0px; border-width:
0px; outline-width: 0px;
vertical-align: baseline;
background-color:
transparent;">
<font face="Ubuntu,
sans-serif" color="#444444"><span
style="line-height: 16px;">Number
of Bricks: 5 x 2 = 10</span></font>
<div style="margin: 0px;
padding: 0px; border-width:
0px; outline-width: 0px;
vertical-align: baseline;
background-color:
transparent;">
<font face="Ubuntu,
sans-serif"
color="#444444"><span
style="line-height:
16px;">Transport-type:
tcp</span></font><br>
<div style="margin: 0px;
padding: 0px;
border-width: 0px;
outline-width: 0px;
vertical-align: baseline;
background-color:
transparent;">
<font face="Ubuntu,
sans-serif"
color="#444444"><span
style="line-height:
16px;">Options
Reconfigured:</span></font></div>
<div style="margin: 0px;
padding: 0px;
border-width: 0px;
outline-width: 0px;
vertical-align: baseline;
background-color:
transparent;">
<font face="Ubuntu,
sans-serif"
color="#444444"><span
style="line-height:
16px;">performance.cache-size:
3GB</span></font></div>
<div style="margin: 0px;
padding: 0px;
border-width: 0px;
outline-width: 0px;
vertical-align: baseline;
background-color:
transparent;">
<font face="Ubuntu,
sans-serif"
color="#444444"><span
style="line-height:
16px;">performance.cache-max-file-size:
512KB</span></font></div>
<div style="margin: 0px;
padding: 0px;
border-width: 0px;
outline-width: 0px;
vertical-align: baseline;
background-color:
transparent;">
<font face="Ubuntu,
sans-serif"
color="#444444"><span
style="line-height:
16px;">network.frame-timeout:
30</span></font></div>
<div style="margin: 0px;
padding: 0px;
border-width: 0px;
outline-width: 0px;
vertical-align: baseline;
background-color:
transparent;">
<font face="Ubuntu,
sans-serif"
color="#444444"><span
style="line-height:
16px;">network.ping-timeout:
25</span></font></div>
<div style="margin: 0px;
padding: 0px;
border-width: 0px;
outline-width: 0px;
vertical-align: baseline;
background-color:
transparent;">
<font face="Ubuntu,
sans-serif"
color="#444444"><span
style="line-height:
16px;">cluster.min-free-disk:
10%</span></font></div>
<font face="Ubuntu,
sans-serif"
color="#444444"><span
style="line-height:
16px;">
<div style="margin: 0px;
padding: 0px;
border-width: 0px;
outline-width: 0px;
vertical-align:
baseline;
background-color:
transparent;">
<font face="Ubuntu,
sans-serif"
color="#444444"><span
style="line-height:
16px;"><br>
</span></font></div>
Our device：</span></font></div>
<div style="color: rgb(68, 68,
68); font-family:
Ubuntu,sans-serif;
font-size: 13px;
line-height: 16px; margin:
0px; padding: 0px;
border-width: 0px;
outline-width: 0px;
vertical-align: baseline;
background-color:
transparent;">
<br>
Dell R710<br>
600Gsas *6<br>
3*8Gmem<br>
<br>
The error info:<br>
<br>
<div style="margin: 0px;
padding: 0px;
border-width: 0px;
outline-width: 0px;
vertical-align: baseline;
background-color:
transparent;">
[2011-12-14
13:24:10.483812] E
[rdma.c:4813:init]
0-rdma.management: Failed
to initialize IB Device
<div style="margin: 0px;
padding: 0px;
border-width: 0px;
outline-width: 0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:10.483828] E
[rpc-transport.c:742:rpc_transport_load]
0-rpc-transport: 'rdma'
initialization failed
<div style="margin: 0px;
padding: 0px;
border-width: 0px;
outline-width: 0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:10.483841] W
[rpcsvc.c:1288:rpcsvc_transport_create]
0-rpc-service: cannot
create listener,
initing the transport
failed
<div style="margin:
0px; padding: 0px;
border-width: 0px;
outline-width: 0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:11.967621] E
[glusterd-store.c:1820:glusterd_store_retrieve_volume]
0-: Unknown key:
brick-0
<div style="margin:
0px; padding: 0px;
border-width: 0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:11.967665] E
[glusterd-store.c:1820:glusterd_store_retrieve_volume]
0-: Unknown key:
brick-1
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:11.967681]
E
[glusterd-store.c:1820:glusterd_store_retrieve_volume]
0-: Unknown key:
brick-2
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:11.967695]
E
[glusterd-store.c:1820:glusterd_store_retrieve_volume]
0-: Unknown
key: brick-3
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:11.967709]
E
[glusterd-store.c:1820:glusterd_store_retrieve_volume]
0-: Unknown
key: brick-4
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:11.967723]
E
[glusterd-store.c:1820:glusterd_store_retrieve_volume]
0-: Unknown
key: brick-5
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:11.967736]
E
[glusterd-store.c:1820:glusterd_store_retrieve_volume]
0-: Unknown
key: brick-6
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:11.967750]
E
[glusterd-store.c:1820:glusterd_store_retrieve_volume]
0-: Unknown
key: brick-7
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:11.967764]
E
[glusterd-store.c:1820:glusterd_store_retrieve_volume]
0-: Unknown
key: brick-8
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:11.967777]
E
[glusterd-store.c:1820:glusterd_store_retrieve_volume]
0-: Unknown
key: brick-9
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:12.465565]
W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management:
reading from
socket failed.
Error
(Transport
endpoint is
not
connected),
peer (<a
moz-do-not-send="true"
href="http://10.1.1.17:1013" target="_blank">10.1.1.17:1013</a>)
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:12.465623]
W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management:
reading from
socket failed.
Error
(Transport
endpoint is
not
connected),
peer (<a
moz-do-not-send="true"
href="http://10.1.1.8:1013" target="_blank">10.1.1.8:1013</a>)
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:12.465656]
W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management:
reading from
socket failed.
Error
(Transport
endpoint is
not
connected),
peer (<a
moz-do-not-send="true"
href="http://10.1.1.10:1013" target="_blank">10.1.1.10:1013</a>)
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:12.465686]
W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management:
reading from
socket failed.
Error
(Transport
endpoint is
not
connected),
peer (<a
moz-do-not-send="true"
href="http://10.1.1.11:1013" target="_blank">10.1.1.11:1013</a>)
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:12.465716]
W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management:
reading from
socket failed.
Error
(Transport
endpoint is
not
connected),
peer (<a
moz-do-not-send="true"
href="http://10.1.1.125:1013" target="_blank">10.1.1.125:1013</a>)
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:12.633288]
W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management:
reading from
socket failed.
Error
(Transport
endpoint is
not
connected),
peer (<a
moz-do-not-send="true"
href="http://10.1.1.65:1006" target="_blank">10.1.1.65:1006</a>)
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:13.138150]
W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management:
reading from
socket failed.
Error
(Transport
endpoint is
not
connected),
peer (<a
moz-do-not-send="true"
href="http://10.1.1.1:1013" target="_blank">10.1.1.1:1013</a>)
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:13.284665]
W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management:
reading from
socket failed.
Error
(Transport
endpoint is
not
connected),
peer (<a
moz-do-not-send="true"
href="http://10.1.1.3:1013" target="_blank">10.1.1.3:1013</a>)
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:15.790805]
W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management:
reading from
socket failed.
Error
(Transport
endpoint is
not
connected),
peer (<a
moz-do-not-send="true"
href="http://10.1.1.8:1013" target="_blank">10.1.1.8:1013</a>)
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:16.113430]
W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management:
reading from
socket failed.
Error
(Transport
endpoint is
not
connected),
peer (<a
moz-do-not-send="true"
href="http://10.1.1.125:1013" target="_blank">10.1.1.125:1013</a>)
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:16.259040]
W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management:
reading from
socket failed.
Error
(Transport
endpoint is
not
connected),
peer (<a
moz-do-not-send="true"
href="http://10.1.1.10:1013" target="_blank">10.1.1.10:1013</a>)
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:16.392058]
W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management:
reading from
socket failed.
Error
(Transport
endpoint is
not
connected),
peer (<a
moz-do-not-send="true"
href="http://10.1.1.17:1013" target="_blank">10.1.1.17:1013</a>)
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:24:16.429444]
W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management:
reading from
socket failed.
Error
(Transport
endpoint is
not
connected),
peer (<a
moz-do-not-send="true"
href="http://10.1.1.11:1013" target="_blank">10.1.1.11:1013</a>)
<div
style="margin:
0px; padding:
0px;
border-width:
0px;
outline-width:
0px;
vertical-align:
baseline;
background-color:
transparent;">
[2011-12-14
13:26:05.787680]
W
[glusterfsd.c:727:cleanup_and_exit]
(-->/lib64/libc.so.6(clone+0x6d)
[0x37c8ed3c2d]
(-->/lib64/libpthread.so.0
[0x37c96064a7]
(-->/opt/glusterfs/3.2.5/sbin/glusterd(glusterfs_sigwaiter+0x17c)
[0x40477c])))
0-: received
signum (15),
shutting down</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div><br>
</div>
-- <br>
<br>
Regards,<br>
<br>
Cocl<br>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
hi Changliang,<br>
Could you specify which process
crashed. Is it glusterd or glusterfs? Could
you provide the stack trace that is present
in it's respective logfile. I dont see any
stack trace in the logs you have provided.<span><font
color="#888888"><br>
<br>
Pranith<br>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<br>
Regards,<br>
<br>
Cocl<br>
OM manager<br>
19lou Operation & Maintenance Dept<br>
</div>
</blockquote>
</div>
</div>
Could you send the logs of all the machines, we will check
and getback to you.<span class="HOEnZb"><font
color="#888888"><br>
<br>
Pranith<br>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>