Hi Mike,<br><br>Do you observe the same issue with vanilla 3.0.2 (without patch 2659)? Also please find the comments inlined.<br><br><div class="gmail_quote">On Sat, Feb 27, 2010 at 4:40 AM, Mike Terzo <span dir="ltr"><<a href="mailto:mterzo@gmail.com">mterzo@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">I had this on the gluster-users list, but really seems to be a bit<br>
more of a development topic.<br>
<div><div></div><div class="h5"><br>
On Fri, Feb 26, 2010 at 5:22 PM, Mike Terzo <<a href="mailto:mterzo@gmail.com">mterzo@gmail.com</a>> wrote:<br>
> I found this in the archives:<br>
> <a href="http://gluster.org/pipermail/gluster-users/20081027/000523.html" target="_blank">http://gluster.org/pipermail/gluster-users/20081027/000523.html</a><br>
><br>
> I don't see any follow ups with any sort of solution. I'm running<br>
> gluster 3.0.2 with patch 2659 with a 64bit client and 64bit server<br>
> without any issues. I turned up another box running 32 bit client<br>
> pointing to 2 different 64bit servers and I had the same issues listed<br>
> above. My kernels on both boxes clients are 2.6.30 neither have any<br>
> patches applied.<br>
><br>
> Here's my client output<br>
><br>
> pending frames:<br>
> frame : type(1) op(LOOKUP)<br>
> frame : type(1) op(STAT)<br>
> frame : type(1) op(STAT)<br>
><br>
> patchset: v3.0.2<br>
> signal received: 11<br>
> time of crash: 2010-02-26 16:01:08<br>
> configuration details:<br>
> argp 1<br>
> backtrace 1<br>
> dlfcn 1<br>
> fdatasync 1<br>
> libpthread 1<br>
> llistxattr 1<br>
> setfsid 1<br>
> spinlock 1<br>
> epoll.h 1<br>
> xattr.h 1<br>
> st_atim.tv_nsec 1<br>
> package-string: glusterfs 3.0.2<br>
> [0xffffe400]<br>
> /usr/lib/libglusterfs.so.0[0xb7ffa8a4]<br>
> /usr/lib/libglusterfs.so.0(inode_unref+0x39)[0xb7ffb571]<br>
> /usr/lib/libglusterfs.so.0(loc_wipe+0x25)[0xb7feec52]<br>
> /usr/lib/libglusterfs.so.0(call_stub_destroy+0x786)[0xb800208e]<br>
> /usr/lib/libglusterfs.so.0(call_resume+0x73)[0xb80022a8]<br>
> /usr/lib/glusterfs/3.0.2/xlator/performance/io-threads.so(iot_worker_unordered+0x20)[0xb75e0ae3]<br>
> /lib/tls/libpthread.so.0[0xb7fcd1ce]<br>
> /lib/tls/libc.so.6(__clone+0x5e)[0xb7f6290e]<br>
> ---------<br>
> Segmentation fault (core dumped)<br>
><br>
> I've turned off io-threads and the process hasn't cored again. Is<br>
> this a general practice to not mix 32bit client and 64bit servers?<br>
><br>
> Thanks,<br>
> --mike terzo<br>
><br>
<br>
</div></div>With io-threads turned not there, i got a much nicer looking stack trace.<br>
<div class="im"><br>
<br>
pending frames:<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(STAT)<br>
frame : type(1) op(STAT)<br>
<br>
patchset: v3.0.2<br>
signal received: 11<br>
</div>time of crash: 2010-02-26 18:28:58<br>
<div class="im">configuration details:<br>
argp 1<br>
backtrace 1<br>
dlfcn 1<br>
fdatasync 1<br>
libpthread 1<br>
llistxattr 1<br>
setfsid 1<br>
spinlock 1<br>
epoll.h 1<br>
xattr.h 1<br>
st_atim.tv_nsec 1<br>
package-string: glusterfs 3.0.2<br>
[0xffffe400]<br>
</div>/usr/lib/libglusterfs.so.0[0xb7f0b8a4]<br>
/usr/lib/libglusterfs.so.0(inode_unref+0x39)[0xb7f0c571]<br>
/usr/lib/libglusterfs.so.0(loc_wipe+0x25)[0xb7effc52]<br>
/usr/lib/glusterfs/3.0.2/xlator/mount/fuse.so[0xb74db3b0]<br>
/usr/lib/glusterfs/3.0.2/xlator/mount/fuse.so[0xb74e4c19]<br>
/usr/lib/libglusterfs.so.0[0xb7f0261e]<br>
/usr/lib/glusterfs/3.0.2/xlator/performance/write-behind.so(wb_stat_cbk+0x179)[0xb74fe035]<br>
/usr/lib/libglusterfs.so.0[0xb7f0261e]<br>
/usr/lib/glusterfs/3.0.2/xlator/cluster/replicate.so(afr_stat_cbk+0xb9)[0xb751f82a]<br>
/usr/lib/glusterfs/3.0.2/xlator/protocol/client.so(client_stat_cbk+0x1bd)[0xb755b525]<br>
/usr/lib/glusterfs/3.0.2/xlator/protocol/client.so(protocol_client_interpret+0x1e1)[0xb754b85d]<br>
/usr/lib/glusterfs/3.0.2/xlator/protocol/client.so(protocol_client_pollin+0xbe)[0xb754c0af]<br>
/usr/lib/glusterfs/3.0.2/xlator/protocol/client.so(notify+0x204)[0xb754f918]<br>
/usr/lib/libglusterfs.so.0(xlator_notify+0x39)[0xb7effae4]<br>
/usr/lib/glusterfs/3.0.2/transport/socket.so(socket_event_poll_in+0x39)[0xb6cd1f96]<br>
/usr/lib/glusterfs/3.0.2/transport/socket.so(socket_event_handler+0x52)[0xb6cd3c6e]<br>
/usr/lib/libglusterfs.so.0[0xb7f1962e]<br>
/usr/lib/libglusterfs.so.0(event_dispatch+0x21)[0xb7f199ae]<br>
glusterfs(main+0xc92)[0x804bcc7]<br>
/lib/tls/libc.so.6(__libc_start_main+0xd4)[0xb7dbeea4]<br>
glusterfs[0x8049e21]<br>
<br>
here's the backtrace of the core:<br>
(gdb) bt<br>
#0 __inode_invalidate (inode=0x805dd14) at inode.c:993<br>
#1 0xb7f0b8a4 in inode_table_prune (table=0x805dd18) at inode.c:1022<br>
#2 0xb7f0c571 in inode_unref (inode=0x805d858) at inode.c:399<br>
#3 0xb7effc52 in loc_wipe (loc=0xb4889aa4) at xlator.c:995<br>
#4 0xb74db3b0 in free_state (state=0xb4889a98) at fuse-bridge.c:182<br>
#5 0xb74e4c19 in fuse_attr_cbk (frame=0xb4892b7c, cookie=0xb04162e0,<br>
this=0x8052248, op_ret=0, op_errno=0,<br>
buf=0xbfd15c9c) at fuse-bridge.c:731<br>
#6 0xb7f0261e in default_stat_cbk (frame=0xb04162e0,<br>
cookie=0xb0667c48, this=0x8058310, op_ret=0, op_errno=0, buf=0x0)<br>
at defaults.c:88<br>
#7 0xb74fe035 in wb_stat_cbk (frame=0xb0667c48, cookie=0xb046bf00,<br>
this=0x8057dd8, op_ret=0, op_errno=0, buf=0x0)<br>
at write-behind.c:543<br>
#8 0xb7f0261e in default_stat_cbk (frame=0xb046bf00,<br>
cookie=0xb06bdd28, this=0x8057828, op_ret=0, op_errno=0, buf=0x0)<br>
at defaults.c:88<br>
#9 0xb751f82a in afr_stat_cbk (frame=0xb06bdd28, cookie=0x0,<br>
this=0x0, op_ret=0, op_errno=0, buf=0xbfd15c9c)<br>
at afr-inode-read.c:227<br>
#10 0xb755b525 in client_stat_cbk (frame=0xb04b0b58, hdr=0xb04c5f00,<br>
hdrlen=188, iobuf=0x0) at client-protocol.c:4105<br>
#11 0xb754b85d in protocol_client_interpret (this=0x0,<br>
trans=0x805a1d8, hdr_p=0xb04c5f00 "", hdrlen=188, iobuf=0x0)<br>
at client-protocol.c:6511<br>
#12 0xb754c0af in protocol_client_pollin (this=0x0, trans=0x805a1d8)<br>
at client-protocol.c:6809<br>
#13 0xb754f918 in notify (this=0x8056d18, event=2, data=0x805a1d8) at<br>
client-protocol.c:6928<br>
#14 0xb7effae4 in xlator_notify (xl=0x8056d18, event=0, data=0x0) at<br>
xlator.c:928<br>
#15 0xb6cd1f96 in socket_event_poll_in (this=0x805a1d8) at socket.c:729<br>
#16 0xb6cd3c6e in socket_event_handler (fd=8, idx=0, data=0x805a1d8,<br>
poll_in=1, poll_out=0, poll_err=0) at socket.c:829<br>
#17 0xb7f1962e in event_dispatch_epoll (event_pool=0x8051e48) at event.c:804<br>
---Type <return> to continue, or q <return> to quit---<br>
#18 0xb7f199ae in event_dispatch (event_pool=0x1ece) at event.c:975<br>
#19 0x0804bcc7 in main (argc=5, argv=0xbfd16b54) at glusterfsd.c:1413<br>
<br>
and table's address, is way wrong.<br>
(gdb) p * inode<br>
$4 = {table = 0x78, lock = 1, nlookup = 33870112096256, generation =<br>
4294967296, in_attic = 0, ref = 14057,<br>
ino = 578108920368060400, st_mode = 134554184, fd_list = {next =<br>
0x539, prev = 0x805deb8}, dentry_list = {<br>
next = 0x8079608, prev = 0xb489c36c}, hash = {next = 0x805db44,<br>
prev = 0x24bd9}, list = {next = 0x805dd58,<br>
prev = 0x805dd58}, _ctx = 0xfffffac4}<br>
<br>
(gdb) up<br>
#1 0xb7f0b8a4 in inode_table_prune (table=0x805dd18) at inode.c:1022<br>
1022 in inode.c<br>
(gdb) p entry<br>
$5 = (inode_t *) 0x1ece<br>
(gdb) p *entry<br>
Cannot access memory at address 0x1ece<br>
(gdb) p table<br>
$6 = (inode_table_t *) 0x805dd18<br>
(gdb) p *table<br>
$7 = {lock = {__m_reserved = 1, __m_count = 0, __m_owner = 0x1ece,<br>
__m_kind = 0, __m_lock = {__status = 1,<br>
__spinlock = 0}}, hashsize = 14057, name = 0x805d7f0<br>
"fuse/inode", root = 0x805db00, xl = 0x8052248,<br>
lru_limit = 1337, inode_hash = 0x805deb8, name_hash = 0x8079608,<br>
active = {next = 0xb489c36c, prev = 0x805db44},<br>
active_size = 150489, lru = {next = 0x805dd58, prev = 0x805dd58},<br>
lru_size = 4294965956,<br>
lru_callback = 0xb7f0c855 <__inode_invalidate>, purge = {next =<br>
0x805dd68, prev = 0x805dd68}, purge_size = 0, inval = {<br>
next = 0xb489c864, prev = 0xb04c7234}, inval_size = 301000, attic<br>
= {next = 0xb633bf44, prev = 0x8095a64},<br>
attic_size = 16}<br>
<br>
looking at inode.c there's a macro that gets called right before this:<br>
<br>
#define list_entry(ptr, type, member) \<br>
((type *)((char *)(ptr)-(unsigned long)(&((type *)0)->member)))<br>
<br>
(unsigned long) gets used here, which is bad.<br>
<br>
(gdb) p *((inode_t *) 0x805dd58)<br>
$20 = {table = 0x805dd58, lock = 134602072, nlookup =<br>
13254313975044111044, generation = 578111566067916136,<br>
in_attic = 0, ref = 3028928612, ino = 1292788113895988, st_mode =<br>
3056844612, fd_list = {next = 0x8095a64,<br>
prev = 0x10}, dentry_list = {next = 0x21, prev = 0xb42dff0c}, hash<br>
= {next = 0xb42dff0c, prev = 0xb41bcdf8}, list = {<br>
next = 0xb5b7f850, prev = 0xb42dfed8}, _ctx = 0x805ddb0}<br>
<br>
printf("size of: %d\n", sizeof(unsigned long));<br>
on my 32bit system:<br>
size of: 4<br>
64bit system:<br>
size of: 8<br>
<br>
looks like a type conversion is breaking everything.<br></blockquote><div><br>Its not an issue with type casting. Pointers and unsigned long are of same size on the same system. Here we are type casting a pointer on a 32 bit system to unsigned long on the same 32 bit system and both will be of same size.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
--mike terzo<br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div><br><br clear="all">regards,<br>-- <br>Raghavendra G<br><br>