<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<STYLE type=text/css> <!--@import url(scrollbar.css); --></STYLE>
<META content="text/html; charset=utf-8" http-equiv=Content-Type>
<STYLE>                        BLOCKQUOTE{margin-Top: 0px; margin-Bottom: 0px; margin-Left: 2em}                        body{FONT-SIZE:12.1pt; COLOR:#001; FONT-FAMILY:宋体,serif;}                </STYLE>
<META name=GENERATOR content="MSHTML 8.00.6001.19190"><BASE
target=_blank></HEAD>
<BODY
style="LINE-HEIGHT: 1.3; BORDER-RIGHT-WIDTH: 0px; MARGIN: 12px; BORDER-TOP-WIDTH: 0px; BORDER-BOTTOM-WIDTH: 0px; BORDER-LEFT-WIDTH: 0px"
marginheight="0" marginwidth="0"><STATIONERY>
<DIV><FONT color=#000000 size=3 face=宋体>Hi all,</FONT></DIV>
<DIV><FONT color=#000000 size=3></FONT> </DIV>
<DIV><FONT color=#000000 size=3>These days, our glusterfs system crashed many
times. The coredump showed the crash point always in FRAME_DESTORY or
STACK_WIND. Once frame->next->prev was referenced, even if the value check
such as 'if (frame->next->prev == NULL) {return}' always
crashed. This mainly occured when opendir was called. There are about 500
bricks in thes system. The coredump listed as follows, and anybody can
help me ? </FONT><FONT color=#000000 size=3></FONT></DIV>
<DIV><FONT color=#000000 size=3>Coredump 1:</FONT></DIV>
<DIV><FONT color=#000000 size=3></FONT> </DIV>
<DIV>#0 0x00002aad0cd465f4 in FRAME_DESTROY (frame=<value optimized
out>, cookie=<value optimized out>, <BR>
this=<value optimized out>, op_ret=<value optimized out>,
op_errno=<value optimized out>, fd=<value optimized
out>)<BR> at
../../../../libglusterfs/src/stack.h:143<BR>143
frame->next->prev = frame->prev;<BR>(gdb) bt<BR>#0
0x00002aad0cd465f4 in FRAME_DESTROY (frame=<value optimized out>,
cookie=<value optimized out>, <BR> this=<value
optimized out>, op_ret=<value optimized out>, op_errno=<value
optimized out>, fd=<value optimized out>)<BR> at
../../../../libglusterfs/src/stack.h:143<BR>#1 STACK_DESTROY
(frame=<value optimized out>, cookie=<value optimized out>,
this=<value optimized out>, <BR> op_ret=<value
optimized out>, op_errno=<value optimized out>, fd=<value optimized
out>)<BR> at
../../../../libglusterfs/src/stack.h:180<BR>#2 fuse_fd_cbk
(frame=<value optimized out>, cookie=<value optimized out>,
this=<value optimized out>, <BR> op_ret=<value
optimized out>, op_errno=<value optimized out>, fd=<value optimized
out>) at fuse-bridge.c:594<BR>#3 0x00002aad0e34e7a9 in
io_stats_opendir_cbk (frame=0x2aadc5b329c0, cookie=<value optimized out>,
<BR> this=<value optimized out>, op_ret=0, op_errno=117,
fd=0x2aad67f98954) at io-stats.c:1492<BR>#4 0x00002aad0e12eab2 in
sp_fd_cbk (frame=0x2aadc5bc0fc0, cookie=<value optimized out>,
this=<value optimized out>, <BR> op_ret=0, op_errno=117,
fd=0x2aad67f98954) at stat-prefetch.c:1506<BR>#5 0x00002aad0df00238 in
dht_fd_cbk (frame=0x2aadc5bc10c0, cookie=<value optimized out>,
this=<value optimized out>, <BR> op_ret=<value
optimized out>, op_errno=<value optimized out>, fd=<value optimized
out>) at dht-common.c:2615<BR>#6 0x00002aad0dc8f941 in
afr_examine_dir_readdir_cbk (frame=0x2aadc3a59ce0, cookie=<value optimized
out>, <BR> this=<value optimized out>,
op_ret=<value optimized out>, op_errno=<value optimized out>,
entries=<value optimized out>)<BR> at
afr-dir-read.c:185<BR>#7 0x00002aad0da6485d in client3_1_readdir_cbk
(req=<value optimized out>, iov=<value optimized out>,
<BR> count=<value optimized out>, myframe=0x36a4600) at
client3_1-fops.c:1883<BR>#8 0x00002aad0a317315 in rpc_clnt_handle_reply
(clnt=0x18a2b30, pollin=0x24054b0) at rpc-clnt.c:741<BR>#9
0x00002aad0a317569 in rpc_clnt_notify (trans=<value optimized out>,
mydata=0x18a2b60, event=<value optimized out>, <BR>
data=<value optimized out>) at rpc-clnt.c:854<BR>#10 0x00002aad0a312418 in
rpc_transport_notify (this=<value optimized out>, event=<value
optimized out>, <BR> data=<value optimized out>) at
rpc-transport.c:919<BR>#11 0x00002aad0d41f254 in socket_event_poll_in
(this=0x18a2c50) at socket.c:1647<BR>#12 0x00002aad0d41f337 in
socket_event_handler (fd=<value optimized out>, idx=405, data=0x18a2c50,
poll_in=1, poll_out=0, <BR> poll_err=<value optimized
out>) at socket.c:1762<BR>#13 0x00002aad0a0e3014 in
event_dispatch_epoll_handler (event_pool=0x1602330) at event.c:794<BR>#14
event_dispatch_epoll (event_pool=0x1602330) at event.c:856<BR>#15
0x0000000000405e69 in main (argc=5, argv=0x7fff4b224d28) at
glusterfsd.c:1462</DIV>
<DIV> </DIV>
<DIV>Coredump 2:</DIV>
<DIV>#0 CHECK_FRAME (frame=0x2af752e9bba0, this=<value optimized
out>, loc=0x2af7538acd38, fd=0x2af6b71f6b4c)<BR> at
../../../../libglusterfs/src/stack.h:205<BR>205
if (frame->root->frames.next->prev == NULL){<BR>(gdb)
bt<BR>#0 CHECK_FRAME (frame=0x2af752e9bba0, this=<value optimized
out>, loc=0x2af7538acd38, fd=0x2af6b71f6b4c) <BR> at
../../../../libglusterfs/src/stack.h:205<BR>#1 afr_opendir
(frame=0x2af752e9bba0, this=<value optimized out>, loc=0x2af7538acd38,
fd=0x2af6b71f6b4c)<BR> at afr-dir-read.c:343<BR>#2
0x00002af65d2d036a in dht_opendir (frame=0x2af751f91ec0, this=<value
optimized out>, loc=0x2af7538acd38, <BR> fd=0x2af6b71f6b4c)
at dht-common.c:3092<BR>#3 0x00002af65d4ff86a in sp_opendir
(frame=<value optimized out>, this=0xa1a730, loc=0x2af7538acd38,
fd=0x2af6b71f6b4c)<BR> at stat-prefetch.c:1854<BR>#4
0x00002af65d719c4a in io_stats_opendir (frame=<value optimized out>,
this=0xa1b940, loc=0x2af7538acd38, <BR> fd=0x2af6b71f6b4c) at
io-stats.c:2137<BR>#5 0x00002af65c111540 in fuse_opendir_resume
(state=0x2af7538acd20) at fuse-bridge.c:2011<BR>#6 0x00002af65c0ff512 in
fuse_resolve_and_resume (state=0x2af7538acd20, fn=0x2af65c1113a0
<fuse_opendir_resume>)<BR> at
fuse-resolve.c:763<BR>#7 0x00002af65c10c5bd in fuse_thread_proc
(data=0x7d16d0) at fuse-bridge.c:3223<BR>#8 0x0000003c394077e1 in
start_thread () from /lib64/libpthread.so.0<BR>#9 0x0000003c390e152d in
clone () from /lib64/libc.so.6</DIV>
<DIV> </DIV>
<DIV>Note: CHECK_FRAME was a function I added to showed the crashed line in the
coredump.</DIV>
<DIV><FONT color=#000000 size=3>Thank you very much</FONT></DIV>
<DIV align=left><FONT color=#c0c0c0 size=2
face=Verdana>2012-04-09</FONT></DIV><FONT size=2 face=Verdana>
<HR style="WIDTH: 122px; HEIGHT: 2px" id=SignNameHR align=left SIZE=2>
</FONT>
<DIV><FONT color=#c0c0c0 size=2 face=Verdana><SPAN
id=_FlashSignName></SPAN></FONT></DIV></STATIONERY></BODY></HTML>