<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: Verdana; font-size: 10pt; color: #000000'>Hi,<br><br>We just ran into a problem with the same result: the crash of a client. We use two storage bricks with replication in the client. We stopped the first storage brick, which caused the crash of the client. <br>Please take a look at my log below. Does anyone know why this is caused?<br><br>Best regards Jasper<br><br><br><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">[2009-05-30 21:02:22] E [saved-frames.c:165:saved_frames_unwind] brick1: forced unwinding frame type(1) op(FINODELK)</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">[2009-05-30 21:02:22] E [saved-frames.c:165:saved_frames_unwind] brick1: forced unwinding frame type(1) op(FINODELK)</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">[2009-05-30 21:02:22] D [socket.c:1229:socket_submit] brick1: not connected (priv->connected = 255)</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">[2009-05-30 21:02:22] N [client-protocol.c:6248:notify] brick1: disconnected</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">[2009-05-30 21:02:22] E [socket.c:744:socket_connect_finish] brick1: connection to 172.23.120.210:6996 failed (Connection refused)</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">pending frames:</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">frame : type(1) op(READ)</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">patchset: 5c1d9108c1529a1155963cb1911f8870a674ab5b</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">signal received: 11</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">configuration details:argp 1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">backtrace 1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">db.h 1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">dlfcn 1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">fdatasync 1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">libpthread 1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">llistxattr 1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">setfsid 1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">spinlock 1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">epoll.h 1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">xattr.h 1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">st_atim.tv_nsec 1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">package-string: glusterfs 2.0.1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">/lib64/libc.so.6[0x381e830280]</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">/usr/lib64/glusterfs/2.0.1/xlator/performance/read-ahead.so(ra_readv+0x58)[0x2b8c0d2a1888]</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">/usr/lib64/glusterfs/2.0.1/xlator/cluster/replicate.so(afr_readv+0x173)[0x2b8c0d4b8203]</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">/usr/lib64/libfuse.so.2[0x2b8c0d8fdf39]</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">/usr/lib64/glusterfs/2.0.1/xlator/mount/fuse.so[0x2b8c0d6deffd]</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">/lib64/libpthread.so.0[0x381f806367]</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">/lib64/libc.so.6(clone+0x6d)[0x381e8d2f7d]</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">---------</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">[2009-05-30 21:02:23] N [client-protocol.c:6248:notify] brick1: disconnected</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><br><br><div><div><div><div><div><div><div><div><div><div><font size="1"><span style="color: rgb(102, 102, 102);">Jasper van Wanrooy - Chatventure BV</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);">Technical Manager</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);">T: +31 (0) 6 47 248 722</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);">E: jvanwanrooy@chatventure.nl</span><br style="color: rgb(102, 102, 102);"><span style="color: rgb(102, 102, 102);">W: www.chatventure.nl</span></font><br></div></div></div></div></div></div></div></div></div></div><br><br>----- Original Message -----<br>From: "Vahriç Muhtaryan" <vahric@doruk.net.tr><br>To: "Alpha Electronics" <myitouchs@gmail.com>, gluster-users@gluster.org<br>Sent: Saturday, 30 May, 2009 10:43:12 GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna<br>Subject: Re: [Gluster-users] Could be the bug of Glusterfs? The file system is unstable and hang<br><br>
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="Section1">
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Hello,</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">I was installed new version like you and making test for
something should be or not . We have same configuration but I got differnet
error, I couldn’t create directory or file , “it was giving Invalid
Argument” and I saw that one of server give an error like below , still
testing .... </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">pending frames:</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">frame : type(1) op(WRITE)</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">patchset: 5c1d9108c1529a1155963cb1911f8870a674ab5b</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">signal received: 6</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">configuration details:argp 1</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">backtrace 1</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">db.h 1</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">dlfcn 1</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">fdatasync 1</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">libpthread 1</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">llistxattr 1</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">setfsid 1</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">spinlock 1</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">epoll.h 1</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">xattr.h 1</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">st_atim.tv_nsec 1</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">package-string: glusterfs 2.0.1</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">[0xfa9420]</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">/lib/libc.so.6(abort+0x101)[0x218691]</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">/lib/libc.so.6[0x24f24b]</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">/lib/libc.so.6[0x2570f1]</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">/lib/libc.so.6(cfree+0x90)[0x25abc0]</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">/usr/local/lib/glusterfs/2.0.1/transport/socket.so(__socket_reset+0x3e)[0xc8155e]</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">/usr/local/lib/glusterfs/2.0.1/transport/socket.so(socket_event_poll_err+0x3b)[0xc8303b]</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">/usr/local/lib/glusterfs/2.0.1/transport/socket.so(socket_event_handler+0x8b)[0xc833bb]</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">/usr/local/lib/libglusterfs.so.0[0x9820ca]</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">/usr/local/lib/libglusterfs.so.0(event_dispatch+0x21)[0x980fb1]</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">glusterfsd(main+0xdf3)[0x804b1a3]</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">/lib/libc.so.6(__libc_start_main+0xdc)[0x203e8c]</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">glusterfsd[0x8049911]</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">---------</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span></p>
<div style="border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; border-width: 1pt medium medium; padding: 3pt 0cm 0cm;">
<p class="MsoNormal"><b><span style="font-size: 10pt; font-family: "Tahoma","sans-serif";" lang="EN-US">From:</span></b><span style="font-size: 10pt; font-family: "Tahoma","sans-serif";" lang="EN-US"> gluster-users-bounces@gluster.org
[mailto:gluster-users-bounces@gluster.org] <b>On Behalf Of </b>Alpha
Electronics<br>
<b>Sent:</b> Friday, May 29, 2009 10:32 PM<br>
<b>To:</b> gluster-users@gluster.org<br>
<b>Subject:</b> [Gluster-users] Could be the bug of Glusterfs? The file system
is unstable and hang</span></p>
</div>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal">We are testing the glusterfs before recommending them to
enterprise clients. We found that the file system always hang after running for
about 2 days. after killing the server side process and then restart,
everything goes back to normal.<br>
<br>
Here is the spec and error logged:<br>
GlusterFS version: v2.0.1<br>
<br>
Client volume:<br>
volume brick_1<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-port 7777 # Non-default port<br>
option remote-host server1<br>
option remote-subvolume brick<br>
end-volume<br>
<br>
volume brick_2<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-port 7777 # Non-default port<br>
option remote-host server2<br>
option remote-subvolume brick<br>
end-volume<br>
<br>
volume bricks<br>
type cluster/distribute<br>
subvolumes brick_1 brick_2<br>
end-volume<br>
<br>
Error logged on client side through /var/log/glusterfs.log<br>
[2009-05-29 14:58:55] E [client-protocol.c:292:call_bail] brick_1: bailing out
frame LK(28) frame sent = 2009-05-29 14:28:54. frame-timeout = 1800<br>
[2009-05-29 14:58:55] W [fuse-bridge.c:2284:fuse_setlk_cbk] glusterfs-fuse:
106850788: ERR => -1 (Transport endpoint is not connected)<br>
error logged on server <br>
[2009-05-29 14:59:15] E [client-protocol.c:292:call_bail] brick_2: bailing out
frame LK(28) frame sent = 2009-05-29 14:29:05. frame-timeout = 1800<br>
[2009-05-29 14:59:15] W [fuse-bridge.c:2284:fuse_setlk_cbk] glusterfs-fuse:
106850860: ERR => -1 (Transport endpoint is not connected)<br>
<br>
There is error message logged on server side after 1 hour in /var/log/messages:<br>
May 29 16:04:16 server2 winbindd[3649]: [2009/05/29 16:05:16, 0]
lib/util_sock.c:write_data(564)<br>
May 29 16:04:16 server2 winbindd[3649]: write_data: write failure.
Error = Connection reset by peer<br>
May 29 16:04:16 server2 winbindd[3649]: [2009/05/29 16:05:16, 0]
libsmb/clientgen.c:write_socket(158)<br>
May 29 16:04:16 server2 winbindd[3649]: write_socket: Error writing
104 bytes to socket 18: ERRNO = Connection reset by peer<br>
May 29 16:04:16 server2 winbindd[3649]: [2009/05/29 16:05:16, 0]
libsmb/clientgen.c:cli_send_smb(188)<br>
May 29 16:04:16 server2 winbindd[3649]: Error writing 104 bytes to
client. -1 (Connection reset by peer)<br>
May 29 16:04:16 server2 winbindd[3649]: [2009/05/29 16:05:16, 0]
libsmb/cliconnect.c:cli_session_setup_spnego(859)<br>
May 29 16:04:16 server2 winbindd[3649]: Kinit failed: Cannot
contact any KDC for requested realm</p>
</div>
<p class="MsoNormal"> </p>
</div>
<br>_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
</div></body></html>