<div dir="ltr">Today， when I ran statedump, node22 still crashed and I got the pending frames<br><br>[2014-02-07 13:32:52.356386] E [marker-quota-helper.c:230:mq_dict_set_contribution] (-->/usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/debug/io-stats.so(io_stats_lookup+0x13e) [0x7f81ca048a3e] (-->/usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/features/marker.so(marker_lookup+0x300) [0x7f81ca25e170] (-->/usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/features/marker.so(mq_req_xattr+0x3c) [0x7f81ca2676ec]))) 0-marker: invalid argument: loc->parent<br>
pending frames:<br><br>patchset: git://<a href="http://git.gluster.com/glusterfs.git">git.gluster.com/glusterfs.git</a><br>signal received: 11<br>time of crash: 2014-02-07 13:32:52<br>configuration details:<br>argp 1<br>
backtrace 1<br>
dlfcn 1<br>fdatasync 1<br>libpthread 1<br>llistxattr 1<br>setfsid 1<br>spinlock 1<br>epoll.h 1<br>xattr.h 1<br>st_atim.tv_nsec 1<br>package-string: glusterfs 3.3.0.5rhs_iqiyi_7<br>/lib64/libc.so.6[0x35ef232920]<br>/usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/protocol/server.so(ltable_dump+0x54)[0x7f81c9e130b4]<br>
/usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/protocol/server.so(server_inode+0xc7)[0x7f81c9e13597]<br>/usr/lib64/libglusterfs.so.0(gf_proc_dump_xlator_info+0x112)[0x30630443f2]<br>/usr/lib64/libglusterfs.so.0(gf_proc_dump_info+0x4b2)[0x3063044bc2]<br>
/usr/sbin/glusterfsd(glusterfs_sigwaiter+0xd2)[0x405cc2]<br>/lib64/libpthread.so.0[0x35ef607851]<br>/lib64/libc.so.6(clone+0x6d)[0x35ef2e767d]<br><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jan 30, 2014 at 3:14 PM, Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Could you give us the back trace<br>
<br>
Pranith<br>
----- Original Message -----<br>
> From: "Mingfan Lu" <<a href="mailto:mingfan.lu@gmail.com">mingfan.lu@gmail.com</a>><br>
> To: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>><br>
> Cc: "haiwei.xie-soulinfo" <<a href="mailto:haiwei.xie@soulinfo.com">haiwei.xie@soulinfo.com</a>>, "<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a> List" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
> Sent: Thursday, January 30, 2014 12:43:44 PM<br>
> Subject: Re: [Gluster-users] help, all latency comes from FINODELK<br>
><br>
> When I triedto execute the statedump, the brick server of the BAD node<br>
> crashed.<br>
><br>
><br>
><br>
> On Wed, Jan 29, 2014 at 8:13 PM, Pranith Kumar Karampuri <<br>
> <a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>> wrote:<br>
><br>
> > Could you take statedump of bricks and get that information please.<br>
> > You can use<br>
> > <a href="https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/sect-User_Guide-Monitor_Workload-Performing_Statedump.html" target="_blank">https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/sect-User_Guide-Monitor_Workload-Performing_Statedump.html</a><br>
> > for taking statedumps.<br>
> ><br>
> > Is this same as <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1056276" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1056276</a> that<br>
> > you raised for? If yes could you attach the files to that bug so that I get<br>
> > the notifications immediately.<br>
> ><br>
> > Pranith<br>
> > ----- Original Message -----<br>
> > > From: "Mingfan Lu" <<a href="mailto:mingfan.lu@gmail.com">mingfan.lu@gmail.com</a>><br>
> > > To: "haiwei.xie-soulinfo" <<a href="mailto:haiwei.xie@soulinfo.com">haiwei.xie@soulinfo.com</a>><br>
> > > Cc: "<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a> List" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
> > > Sent: Tuesday, January 28, 2014 7:44:14 AM<br>
> > > Subject: Re: [Gluster-users] help, all latency comes from FINODELK<br>
> > ><br>
> > > About 200+ clients<br>
> > ><br>
> > > How to print print lock info in bricks, print lock requst info in afr &<br>
> > dht?<br>
> > > thanks.<br>
> > ><br>
> > ><br>
> > ><br>
> > > On Tue, Jan 28, 2014 at 9:51 AM, haiwei.xie-soulinfo <<br>
> > > <a href="mailto:haiwei.xie@soulinfo.com">haiwei.xie@soulinfo.com</a> > wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > > hi,<br>
> > > looks FINODELK deak lock. how many clients, and nfs or fuse?<br>
> > > Maybe the best way is to print lock info in bricks, print lock requst<br>
> > info in<br>
> > > afr & dht.<br>
> > ><br>
> > > > all my clients hang when they creating dir<br>
> > > ><br>
> > > ><br>
> > > > On Mon, Jan 27, 2014 at 11:33 PM, Mingfan Lu < <a href="mailto:mingfan.lu@gmail.com">mingfan.lu@gmail.com</a> ><br>
> > > > wrote:<br>
> > > ><br>
> > > > > I ran glusterfs volume profile my_volume info, I got some thing<br>
> > likes:<br>
> > > > ><br>
> > > > > 0.00 0.00 us 0.00 us 0.00 us 30 FORGET<br>
> > > > > 0.00 0.00 us 0.00 us 0.00 us 185<br>
> > > > > RELEASE<br>
> > > > > 0.00 0.00 us 0.00 us 0.00 us 11<br>
> > > > > RELEASEDIR<br>
> > > > > 0.00 66.50 us 54.00 us 79.00 us 2<br>
> > > > > SETATTR<br>
> > > > > 0.00 44.83 us 25.00 us 94.00 us 6<br>
> > > > > READDIR<br>
> > > > > 0.00 57.55 us 28.00 us 95.00 us 11<br>
> > > > > OPENDIR<br>
> > > > > 0.00 325.00 us 279.00 us 371.00 us 2<br>
> > > > > RENAME<br>
> > > > > 0.00 147.80 us 84.00 us 206.00 us 5<br>
> > > > > LINK<br>
> > > > > 0.00 389.50 us 55.00 us 724.00 us 2<br>
> > > > > READDIRP<br>
> > > > > 0.00 164.25 us 69.00 us 287.00 us 8<br>
> > > > > UNLINK<br>
> > > > > 0.00 37.46 us 18.00 us 87.00 us 50<br>
> > > > > FSTAT<br>
> > > > > 0.00 70.32 us 29.00 us 210.00 us 37<br>
> > > > > GETXATTR<br>
> > > > > 0.00 77.75 us 42.00 us 216.00 us 55<br>
> > > > > SETXATTR<br>
> > > > > 0.00 36.39 us 11.00 us 147.00 us 119<br>
> > > > > FLUSH<br>
> > > > > 0.00 51.22 us 24.00 us 139.00 us 275<br>
> > > > > OPEN<br>
> > > > > 0.00 180.14 us 84.00 us 457.00 us 96<br>
> > > > > XATTROP<br>
> > > > > 0.00 3847.20 us 231.00 us 18218.00 us 5<br>
> > > > > MKNOD<br>
> > > > > 0.00 70.08 us 15.00 us 6539.00 us 342<br>
> > > > > ENTRYLK<br>
> > > > > 0.00 10338.86 us 184.00 us 34813.00 us 7<br>
> > > > > CREATE<br>
> > > > > 0.00 896.65 us 12.00 us 83103.00 us 235<br>
> > > > > INODELK<br>
> > > > > 0.00 187.86 us 50.00 us 668.00 us 1526<br>
> > > > > WRITE<br>
> > > > > 0.00 40.66 us 13.00 us 400.00 us 10400<br>
> > > > > STATFS<br>
> > > > > 0.00 313.13 us 66.00 us 2142.00 us 2049<br>
> > > > > FXATTROP<br>
> > > > > 0.00 2794.97 us 26.00 us 78048.00 us 295<br>
> > > > > READ<br>
> > > > > 0.00 24469.82 us 206.00 us 176157.00 us 34<br>
> > > > > MKDIR<br>
> > > > > 0.00 40.49 us 13.00 us 507.00 us 21420<br>
> > > > > STAT<br>
> > > > > 0.00 190.90 us 40.00 us 330032.00 us 45820<br>
> > > > > LOOKUP<br>
> > > > > 100.00 72004815.62 us 8.00 us 5783044563.00 us 3994<br>
> > > > > FINODELK<br>
> > > > ><br>
> > > > > what happend?<br>
> > > > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > Gluster-users mailing list<br>
> > > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> > > <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
> ><br>
><br>
</blockquote></div><br></div>