<html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><style>body { line-height: 1.5; }blockquote { margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em; }body { font-size: 10.5pt; font-family: 微软雅黑; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body>
<div> to Krishnan Parthasarathi:</div><div><span></span><span style="font-family: Tahoma, Arial; font-size: 12px; line-height: 24px; background-color: rgb(242, 242, 242);">Thank you very much for your help, </span><span style="background-color: rgb(242, 242, 242); font-family: Tahoma, Arial; font-size: 12px; line-height: 24px;">Your reply is great help to me, thank you, </span></div><div><span style="background-color: rgb(242, 242, 242); font-family: Tahoma, Arial; font-size: 12px; line-height: 24px;">I </span><span style="background-color: rgb(242, 242, 242); color: rgb(67, 67, 67); font-family: Tahoma, Arial; font-size: 12px; line-height: 24px;">Adopt the method of you, modify probe_glusterfs.stap below:</span></div><div><span style="background-color: rgb(242, 242, 242); color: rgb(67, 67, 67); font-family: Tahoma, Arial; font-size: 12px; line-height: 24px;"><br></span></div><div><span style="background-color: rgb(242, 242, 242); color: rgb(67, 67, 67); font-family: Tahoma, Arial; font-size: 12px; line-height: 24px;">execute command: stap </span><span style="font-size: 10.5pt; line-height: 1.5; background-color: window;">probe_glusterfs.stap -D </span><span style="font-size: 10.5pt; line-height: 1.5; background-color: window;">MAXBACKTRACE=200 </span></div><div><span style="font-family: "" tahoma,="" arial'";="" font-size:="" 12px;="" color:="" rgb(67,="" 67,="" 67);="" background-color:="" rgba(0,="" 0,="" 0);="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><br>#!/usr/bin/evn stap
<br>
<br>probe process("/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so").statement("*")
<br>{
<br> print_ubacktrace();
<br>}
<br>
<br>probe process("/usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so").statement("*")
<br>{
<br> print_ubacktrace();
<br>}
<br>
<br>probe process("/usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so").statement("*")
<br>{
<br> print_ubacktrace();
<br>}
<br>
<br>probe process("/usr/local/lib/glusterfs/3.5/xlator/cluster/pump.so").statement("*")
<br>{
<br> print_ubacktrace();
<br>}
<br>
<br>
<br>probe process("/usr/local/lib/glusterfs/3.5/xlator/cluster/replicate.so").statement("*")
<br>{
<br> print_ubacktrace();
<br>}
<br>
<br>
<br>probe process("/usr/local/lib/glusterfs/3.5/xlator/protocol/client.so").statement("*")
<br>{
<br> print_ubacktrace();
<br>}
<br>
<br>probe process("/usr/local/lib/libgfrpc.so").statement("*")
<br>{
<br> print_ubacktrace();
<br>}
<br>
<br>probe process("/usr/local/lib/glusterfs/3.5/rpc-transport/socket.so").statement("*")
<br>{
<br> print_ubacktrace();
<br>}
<br>
<br>
<br>probe process("/usr/local/lib/libglusterfs.so").statement("*")
<br>{
<br> print_ubacktrace();
<br>}
<br>
<br>
<br>probe process("/usr/local/sbin/glusterfsd").statement("*")
<br>{
<br> print_ubacktrace();
<br>}</span></div>
<div><br></div><hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left">
<div><span><div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt"><div>justglusterfs@gmail.com</div></div></span></div>
<blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div> </div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"><div style="PADDING-RIGHT: 8px; PADDING-LEFT: 8px; FONT-SIZE: 12px;FONT-FAMILY:tahoma;COLOR:#000000; BACKGROUND: #efefef; PADDING-BOTTOM: 8px; PADDING-TOP: 8px"><div><b>From:</b> <a href="mailto:kparthas@redhat.com">Krishnan Parthasarathi</a></div><div><b>Date:</b> 2014-05-14 14:29</div><div><b>To:</b> <a href="mailto:justglusterfs@gmail.com">justglusterfs</a></div><div><b>CC:</b> <a href="mailto:gluster-devel@gluster.org">gluster-devel</a></div><div><b>Subject:</b> Re: [Gluster-devel] gluster IRC and use systemtap can't output glusterfs backtrace</div></div></div><div><div>What do you mean by the back trace is not continuous?</div>
<div> </div>
<div>GlusterFS code is based on asynchronous event based programming built around non-blocking</div>
<div>sockets. So, there is a call and a call back that execute in different threads</div>
<div>of execution in the mount process, for every file operation on the mount.This </div>
<div>could be one of the reasons why the back trace appears discontinuous.</div>
<div> </div>
<div>If you think the backtrace appears truncated then check man stap(1) for increasing</div>
<div>default backtrace size (See MAXBACKTRACE).</div>
<div> </div>
<div>Hope that helps,</div>
<div>Krish</div>
<div> </div>
<div>----- Original Message -----</div>
<div>> to Krishnan Parthasarathi:</div>
<div>> I write a systemtap script（probe_glusterfs.stap）： the script content as</div>
<div>> follows：</div>
<div>> </div>
<div>> probe</div>
<div>> process("/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so").statement("*")</div>
<div>> {</div>
<div>> print_ubacktrace();</div>
<div>> }</div>
<div>> </div>
<div>> probe</div>
<div>> process("/usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so").statement("*")</div>
<div>> {</div>
<div>> print_ubacktrace();</div>
<div>> }</div>
<div>> </div>
<div>> probe</div>
<div>> process("/usr/local/lib/glusterfs/3.5/xlator/protocol/client.so").statement("*")</div>
<div>> {</div>
<div>> print_ubacktrace();</div>
<div>> }</div>
<div>> </div>
<div>> when I do "ls /mnt/glusterfs-volume" test， I execute command: stap</div>
<div>> probe_glusterfs.stap</div>
<div>> systemtap output as follows info:</div>
<div>> but the call stack information is not continuous,</div>
<div>> Actually, I want to watch glusterfs all function call stacks when I execute</div>
<div>> "ls /mnt/glusterfs-volume" or "mkdir /mnt/glusterfs-volume/testdir "</div>
<div>> (pay attention to: /mnt/glusterfs-volume is my glusterfs volume mount dir)</div>
<div>> </div>
<div>> systemtap output as follows info:</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46a6980a0 : fuse_resolve_all+0x0/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697dde : fuse_resolve+0x24e/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46a697b90 : fuse_resolve+0x0/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697dde : fuse_resolve+0x24e/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46a6980a0 : fuse_resolve_all+0x0/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697dde : fuse_resolve+0x24e/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46a698100 : fuse_resolve_all+0x60/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697dde : fuse_resolve+0x24e/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46a69f590 : fuse_readdir_resume+0x0/0x3f0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697dde : fuse_resolve+0x24e/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46a696660 : get_call_frame_for_req+0x0/0x270</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697dde : fuse_resolve+0x24e/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46a69667f : get_call_frame_for_req+0x1f/0x270</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697dde : fuse_resolve+0x24e/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46a6966d3 : get_call_frame_for_req+0x73/0x270</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697dde : fuse_resolve+0x24e/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46a696724 : get_call_frame_for_req+0xc4/0x270</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697dde : fuse_resolve+0x24e/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46a696769 : get_call_frame_for_req+0x109/0x270</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697dde : fuse_resolve+0x24e/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46a6964a0 : frame_fill_groups+0x0/0x1c0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69681a : get_call_frame_for_req+0x1ba/0x270</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a697dde : fuse_resolve+0x24e/0x510</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb4675d2c60 : afr_readdirp+0x0/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb4675d25f0 : afr_do_readdir+0x0/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb46760bea0 : afr_local_init+0x0/0x100</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2699 : afr_do_readdir+0xa9/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb467609990 : afr_up_children_count+0x0/0x10</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760bf06 : afr_local_init+0x66/0x100</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2699 : afr_do_readdir+0xa9/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb467609960 : afr_set_elem_count_get+0x0/0x30</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760bf06 : afr_local_init+0x66/0x100</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2699 : afr_do_readdir+0xa9/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb46760b170 : afr_children_create+0x0/0x80</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d26aa : afr_do_readdir+0xba/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb467612f00 : afr_inode_get_read_ctx+0x0/0x40</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d26ca : afr_do_readdir+0xda/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb467612d70 : afr_inode_get_ctx_params+0x0/0x190</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb467612f29 : afr_inode_get_read_ctx+0x29/0x40</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d26ca : afr_do_readdir+0xda/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb467612820 : __afr_inode_ctx_get+0x0/0x110</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb467612dc7 : afr_inode_get_ctx_params+0x57/0x190</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb467612f29 : afr_inode_get_read_ctx+0x29/0x40</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d26ca : afr_do_readdir+0xda/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb46760b550 : afr_get_call_child+0x0/0x260</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d26f1 : afr_do_readdir+0x101/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb4675ec2b0 : afr_fd_ctx_get+0x0/0x60</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2704 : afr_do_readdir+0x114/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb4675ec1e0 : __afr_fd_ctx_get+0x0/0xd0</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675ec2e0 : afr_fd_ctx_get+0x30/0x60</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2704 : afr_do_readdir+0x114/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb4678389f0 : client_readdirp+0x0/0x1d0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb4675d2889 : afr_do_readdir+0x299/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb467843420 : client3_3_readdirp+0x0/0x4c0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467838a79 : client_readdirp+0x89/0x1d0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb4675d2889 : afr_do_readdir+0x299/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb46783e570 : this_fd_get_ctx+0x0/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb4678435d0 : client3_3_readdirp+0x1b0/0x4c0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467838a79 : client_readdirp+0x89/0x1d0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb4675d2889 : afr_do_readdir+0x299/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb46783cee0 : client_submit_request+0x0/0x4c0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467843743 : client3_3_readdirp+0x323/0x4c0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467838a79 : client_readdirp+0x89/0x1d0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb4675d2889 : afr_do_readdir+0x299/0x670</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4673a14b9</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]</div>
<div>> 0x7fb46a6acf40 : fuse_graph_sync+0x0/0x170</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad2e0 : fuse_thread_proc+0x230/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46784b2b0 : client3_3_readdirp_cbk+0x0/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb4675d2ca0 : afr_readdirp_cbk+0x0/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46a6ac8c0 : fuse_readdir_cbk+0x0/0x430</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46673f12e</div>
<div>> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]</div>
<div>> 0x7fb46a698f70 : send_fuse_data+0x0/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6acabd : fuse_readdir_cbk+0x1fd/0x430</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46673f12e</div>
<div>> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]</div>
<div>> 0x7fb46a698da0 : send_fuse_iov+0x0/0x1d0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a698fa6 : send_fuse_data+0x36/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6acabd : fuse_readdir_cbk+0x1fd/0x430</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46673f12e</div>
<div>> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]</div>
<div>> 0x7fb46a696a30 : free_fuse_state+0x0/0x150</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6acac5 : fuse_readdir_cbk+0x205/0x430</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46673f12e</div>
<div>> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]</div>
<div>> 0x7fb46a6969c0 : fuse_resolve_wipe+0x0/0x70</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a696afe : free_fuse_state+0xce/0x150</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6acac5 : fuse_readdir_cbk+0x205/0x430</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46673f12e</div>
<div>> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]</div>
<div>> 0x7fb46a6969c0 : fuse_resolve_wipe+0x0/0x70</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a696b0a : free_fuse_state+0xda/0x150</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6acac5 : fuse_readdir_cbk+0x205/0x430</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46673f12e</div>
<div>> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]</div>
<div>> 0x7fb46a6acac9 : fuse_readdir_cbk+0x209/0x430</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46673f12e</div>
<div>> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]</div>
<div>> 0x7fb46a6acad6 : fuse_readdir_cbk+0x216/0x430</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46673f12e</div>
<div>> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]</div>
<div>> 0x7fb46760dfc0 : afr_local_cleanup+0x0/0x270</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46a699bc0 : fuse_releasedir+0x0/0x1e0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46760ddd0 : afr_local_sh_cleanup+0x0/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760dfda : afr_local_cleanup+0x1a/0x270</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46a6968d0 : get_fuse_state+0x0/0xf0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a699bf4 : fuse_releasedir+0x34/0x1e0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46760d600 : afr_xattr_array_destroy+0x0/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760de18 : afr_local_sh_cleanup+0x48/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760dfda : afr_local_cleanup+0x1a/0x270</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46a696938 : get_fuse_state+0x68/0xf0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a699bf4 : fuse_releasedir+0x34/0x1e0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46760d5c0 : afr_reset_xattr+0x0/0x40</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760d609 : afr_xattr_array_destroy+0x9/0x20</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760de18 : afr_local_sh_cleanup+0x48/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760dfda : afr_local_cleanup+0x1a/0x270</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760de33 : afr_local_sh_cleanup+0x63/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760dfda : afr_local_cleanup+0x1a/0x270</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760de42 : afr_local_sh_cleanup+0x72/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760dfda : afr_local_cleanup+0x1a/0x270</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46760df10 : afr_local_transaction_cleanup+0x0/0xb0</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760dfe5 : afr_local_cleanup+0x25/0x270</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760df2f : afr_local_transaction_cleanup+0x1f/0xb0</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760dfe5 : afr_local_cleanup+0x25/0x270</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760df3e : afr_local_transaction_cleanup+0x2e/0xb0</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760dfe5 : afr_local_cleanup+0x25/0x270</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb467605120 : afr_entry_lockee_cleanup+0x0/0x70</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760df6e : afr_local_transaction_cleanup+0x5e/0xb0</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760dfe5 : afr_local_cleanup+0x25/0x270</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46760e0ff : afr_local_cleanup+0x13f/0x270</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46a699330 : send_fuse_err+0x0/0x140</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a699d0a : fuse_releasedir+0x14a/0x1e0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46a696120 : fuse_ino_to_inode+0x0/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69938a : send_fuse_err+0x5a/0x140</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a699d0a : fuse_releasedir+0x14a/0x1e0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46783e510 : client_local_wipe+0x0/0x60</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46784b45f : client3_3_readdirp_cbk+0x1af/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46a696138 : fuse_ino_to_inode+0x18/0x40</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a69938a : send_fuse_err+0x5a/0x140</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a699d0a : fuse_releasedir+0x14a/0x1e0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb4678377d0 : client_releasedir+0x0/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb467847ab0 : client3_3_releasedir+0x0/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb46a698da0 : send_fuse_iov+0x0/0x1d0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6993fb : send_fuse_err+0xcb/0x140</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a699d0a : fuse_releasedir+0x14a/0x1e0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46a696a30 : free_fuse_state+0x0/0x150</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a699d12 : fuse_releasedir+0x152/0x1e0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46783e850 : this_fd_del_ctx+0x0/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467847b0d : client3_3_releasedir+0x5d/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb46a6969c0 : fuse_resolve_wipe+0x0/0x70</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a696afe : free_fuse_state+0xce/0x150</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a699d12 : fuse_releasedir+0x152/0x1e0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb467847b1c : client3_3_releasedir+0x6c/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb46a6969c0 : fuse_resolve_wipe+0x0/0x70</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a696b0a : free_fuse_state+0xda/0x150</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a699d12 : fuse_releasedir+0x152/0x1e0</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb467847530 : client_fdctx_destroy+0x0/0x4b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb46a6acf40 : fuse_graph_sync+0x0/0x170</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x7fb46a6ad2e0 : fuse_thread_proc+0x230/0x660</div>
<div>> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]</div>
<div>> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]</div>
<div>> 0x7fb46784775f : client_fdctx_destroy+0x22f/0x4b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb46784779b : client_fdctx_destroy+0x26b/0x4b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb46783cee0 : client_submit_request+0x0/0x4c0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46784788f : client_fdctx_destroy+0x35f/0x4b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb4675d24e0 : afr_releasedir+0x0/0x110</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb4675d24e2 : afr_releasedir+0x2/0x110</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb46760bbb0 : afr_cleanup_fd_ctx+0x0/0xe0</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb4675d259d : afr_releasedir+0xbd/0x110</div>
<div>> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb4678377d0 : client_releasedir+0x0/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb467847ab0 : client3_3_releasedir+0x0/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb46783e850 : this_fd_del_ctx+0x0/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467847b0d : client3_3_releasedir+0x5d/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb467847b1c : client3_3_releasedir+0x6c/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb467847530 : client_fdctx_destroy+0x0/0x4b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb46784775f : client_fdctx_destroy+0x22f/0x4b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb46784779b : client_fdctx_destroy+0x26b/0x4b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb46783cee0 : client_submit_request+0x0/0x4c0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46784788f : client_fdctx_destroy+0x35f/0x4b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb467837823 : client_releasedir+0x53/0xe0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]</div>
<div>> 0x7fb46783e050 : clnt_readdirp_rsp_cleanup+0x0/0x50</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46784b498 : client3_3_readdirp_cbk+0x1e8/0x3b0</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46784ce70 : client3_3_releasedir_cbk+0x0/0x120</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46784ce7d : client3_3_releasedir_cbk+0xd/0x120</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46784ce8b : client3_3_releasedir_cbk+0x1b/0x120</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46784ce70 : client3_3_releasedir_cbk+0x0/0x120</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46784ce7d : client3_3_releasedir_cbk+0xd/0x120</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> 0x7fb46784ce8b : client3_3_releasedir_cbk+0x1b/0x120</div>
<div>> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]</div>
<div>> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]</div>
<div>> </div>
<div>> </div>
<div>> </div>
<div>> </div>
<div>> justglusterfs@gmail.com</div>
<div>> </div>
<div>> From: Krishnan Parthasarathi</div>
<div>> Date: 2014-05-14 11:26</div>
<div>> To: justglusterfs</div>
<div>> CC: gluster-devel</div>
<div>> Subject: Re: [Gluster-devel] gluster IRC and use systemtap can't output</div>
<div>> glusterfs backtrace</div>
<div>> This systemtap script limits itself to the fuse module and glusterfs binary.</div>
<div>> But it is possible to track all the modules that can be possibly</div>
<div>> (dynamically)</div>
<div>> loaded into a glusterfs process using system scripts.</div>
<div>> </div>
<div>> I am not sure what kind of analysis you are looking at, across all function</div>
<div>> call stacks.</div>
<div>> </div>
<div>> thanks,</div>
<div>> Krish</div>
<div>> </div>
<div>> ----- Original Message -----</div>
<div>> > to Krishnan Parthasarathi:</div>
<div>> > thank you very much for your help,</div>
<div>> > As you said, I use stap -d /usr/local/sbin/glusterfs --ldd -e 'probe</div>
<div>> > process("/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so").function("*"){print_ubacktrace()}'</div>
<div>> > ,</div>
<div>> > systemtap can output the fuse module function of the stack ,</div>
<div>> > But now I want to track glusterfs all function'call in order to analyze</div>
<div>> > glusterfs system, need output all function call stack ,</div>
<div>> > Do you have any better suggestion? or Recommend other tools（similar to</div>
<div>> > systemtap）?</div>
<div>> > thank you very much!</div>
<div>> > </div>
<div>> > </div>
<div>> > </div>
<div>> > </div>
<div>> > justglusterfs@gmail.com</div>
<div>> > </div>
<div>> > From: Krishnan Parthasarathi</div>
<div>> > Date: 2014-05-13 13:38</div>
<div>> > To: 陈陈</div>
<div>> > CC: gluster-devel</div>
<div>> > Subject: Re: [Gluster-devel] gluster IRC and use systemtap can't output</div>
<div>> > glusterfs backtrace</div>
<div>> > To probe functions in specific translators you need to</div>
<div>> > specify the path to the corresponding .so file in the process</div>
<div>> > provider.</div>
<div>> > </div>
<div>> > For eg, to probe functions in fuse translator,</div>
<div>> > </div>
<div>> > stap -d /usr/local/sbin/glusterfs --ldd -e 'probe</div>
<div>> > process("/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so").function("*"){print_ubacktrace()}'</div>
<div>> > 确实可以输出fuse模块的函数栈</div>
<div>> > The following is the output I see, when I write files using dd on</div>
<div>> > a GlusterFS native mount,</div>
<div>> > </div>
<div>> > <snip></div>
<div>> > </div>
<div>> > 0x31d7cf5e1d : clone+0x6d/0x90 [/usr/lib64/libc-2.17.so]</div>
<div>> > 0x7f7977a1c834 : fuse_resolve+0x74/0x4f0</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x7f7977a1cd3e : fuse_resolve_all+0x8e/0xb0</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x7f7977a1cd80 : fuse_resolve_and_resume+0x20/0x30</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x7f7977a3507e : fuse_thread_proc+0x2ae/0x690</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x31d8407c53 : start_thread+0xc3/0x310 [/usr/lib64/libpthread-2.17.so]</div>
<div>> > 0x31d7cf5e1d : clone+0x6d/0x90 [/usr/lib64/libc-2.17.so]</div>
<div>> > 0x7f7977a1e4d0 : fuse_fd_ctx_get+0x0/0x40</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x7f7977a1c84c : fuse_resolve+0x8c/0x4f0</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x7f7977a1cd3e : fuse_resolve_all+0x8e/0xb0</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x7f7977a1cd80 : fuse_resolve_and_resume+0x20/0x30</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x7f7977a3507e : fuse_thread_proc+0x2ae/0x690</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x31d8407c53 : start_thread+0xc3/0x310 [/usr/lib64/libpthread-2.17.so]</div>
<div>> > 0x31d7cf5e1d : clone+0x6d/0x90 [/usr/lib64/libc-2.17.so]</div>
<div>> > 0x7f7977a1c1e0 : fuse_resolve_continue+0x0/0x100</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x7f7977a1c9f8 : fuse_resolve+0x238/0x4f0</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x7f7977a1cd3e : fuse_resolve_all+0x8e/0xb0</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x7f7977a1cd80 : fuse_resolve_and_resume+0x20/0x30</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x7f7977a3507e : fuse_thread_proc+0x2ae/0x690</div>
<div>> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]</div>
<div>> > 0x31d8407c53 : start_thread+0xc3/0x310 [/usr/lib64/libpthread-2.17.so]</div>
<div>> > 0x31d7cf5e1d : clone+0x6d/0x90 [/usr/lib64/libc-2.17.so]</div>
<div>> > </div>
<div>> > </snip></div>
<div>> > </div>
<div>> > Hope that helps,</div>
<div>> > Krish</div>
<div>> > </div>
<div>> > </div>
<div>> > ----- Original Message -----</div>
<div>> > > See inline for response.</div>
<div>> > > </div>
<div>> > > ----- Original Message -----</div>
<div>> > > > hi everyone!</div>
<div>> > > > I meet three question:</div>
<div>> > > > </div>
<div>> > > > question one:</div>
<div>> > > > </div>
<div>> > > > I use IRC client(mirc) config server is: irc.freenode.net or</div>
<div>> > > > irc.gnu.org</div>
<div>> > > > but always can't connect, follwing errors:</div>
<div>> > > > Connect retry #1 irc,freenode,net (6667) (dns pool)</div>
<div>> > > > unable to connect server (connect timed out)</div>
<div>> > > > </div>
<div>> > > > </div>
<div>> > > > </div>
<div>> > > > question two:</div>
<div>> > > > I system is centos6.4，glusterfs version is 3.4.</div>
<div>> > > > I want to use systemtap watch glusterfs function backtrace,</div>
<div>> > > > when I configure glusterfs source code, execute the following</div>
<div>> > > > operation:</div>
<div>> > > > ./configure --enable--systemtap</div>
<div>> > > > but checking dtrace happen error，because in my system，no dtrace，my</div>
<div>> > > > question is --enable--systemtap why need dtrace? it must have dtrace?</div>
<div>> > > </div>
<div>> > > I think the systemtap folks have retained certain symbols from dtrace</div>
<div>> > > so that code instrumented with systemtap user specified probes are</div>
<div>> > > compatible</div>
<div>> > > with systems that have dtrace but not systemtap in them.</div>
<div>> > > </div>
<div>> > > > </div>
<div>> > > > question three:</div>
<div>> > > > when I do io test in glusterfs client. I execute following command:</div>
<div>> > > > stap -d /usr/local/sbin/glusterfs --ldd -e 'probe</div>
<div>> > > > process("glusterfs").function("*"){print_ubacktrace()}'</div>
<div>> > > > </div>
<div>> > > > the result is there is nothing output, and the stap command S state,</div>
<div>> > > > no return.</div>
<div>> > > </div>
<div>> > > For 'process' systemtap provider, you don't need to compile glusterfs</div>
<div>> > > with</div>
<div>> > > "--enable-systemtap". I am not sure why you are not seeing any of the</div>
<div>> > > probes</div>
<div>> > > firing in your stap script. Let me try this same and share my results on</div>
<div>> > > this</div>
<div>> > > thread.</div>
<div>> > > </div>
<div>> > > Hope that helps,</div>
<div>> > > Krish</div>
<div>> > > </div>
<div>> > > > </div>
<div>> > > > thank you answer me!</div>
<div>> > > > _______________________________________________</div>
<div>> > > > Gluster-devel mailing list</div>
<div>> > > > Gluster-devel@gluster.org</div>
<div>> > > > http://supercolony.gluster.org/mailman/listinfo/gluster-devel</div>
<div>> > > > </div>
<div>> > > _______________________________________________</div>
<div>> > > Gluster-devel mailing list</div>
<div>> > > Gluster-devel@gluster.org</div>
<div>> > > http://supercolony.gluster.org/mailman/listinfo/gluster-devel</div>
<div>> > > </div>
<div>> > </div>
<div>> </div>
</div></blockquote>
</body></html>