<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">There are a few suspicious things going on here..</div><div class="gmail_quote"><br></div><div class="gmail_quote">On Tue, May 20, 2014 at 10:07 PM, Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
> > hi,<br>
> > crypt.t is failing regression builds once in a while and most of<br>
> > the times it is because of the failures just after the remount in the<br>
> > script.<br>
> ><br>
> > TEST rm -f $M0/testfile-symlink<br>
> > TEST rm -f $M0/testfile-link<br>
> ><br>
> > Both of these are failing with ENOTCONN. I got a chance to look at<br>
> > the logs. According to the brick logs, this is what I see:<br>
> > [2014-05-17 05:43:43.363979] E [posix.c:2272:posix_open]<br>
> > 0-patchy-posix: open on /d/backends/patchy1/testfile-symlink:<br>
> > Transport endpoint is not connected<br></div></div></blockquote><div><br></div><div>posix_open() happening on a symlink? This should NEVER happen. glusterfs itself should NEVER EVER by triggering symlink resolution on the server. In this case, for whatever reason an open() is attempted on a symlink, and it is getting followed back onto gluster's own mount point (test case is creating an absolute link).</div>
<div><br></div><div>So first find out: who is triggering fop->open() on a symlink. Fix the caller.</div><div><br></div><div>Next: add a check in posix_open() to fail with ELOOP or EINVAL if the inode is a symlink.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
> ><br>
> > This is the very first time I saw posix failing with ENOTCONN. Do we<br>
> > have these bricks on some other network mounts? I wonder why it fails<br>
> > with ENOTCONN.<br>
> ><br>
> > I also see that it happens right after a call_bail on the mount.<br>
> ><br>
> > Pranith<br>
><br>
> Hello.<br>
> OK, I'll try to reproduce it.<br>
<br>
</div></div>I tried re-creating the issue on my fedora VM and it happened just now. When this issue happens I am not able to attach the process to gdb. From /proc/ the threads are in the following state for a while now:<br>
root@pranith-vm1 - /proc/4053/task<br>
10:20:50 :) ⚡ for i in `ls`; do cat $i/stack; echo "---------------------------------"; done<br>
[<ffffffff811ed8ce>] ep_poll+0x21e/0x330<br>
[<ffffffff811ee7b5>] SyS_epoll_wait+0xd5/0x100<br>
[<ffffffff816533d9>] system_call_fastpath+0x16/0x1b<br>
[<ffffffffffffffff>] 0xffffffffffffffff<br>
---------------------------------<br>
[<ffffffff8108cb6d>] hrtimer_nanosleep+0xad/0x170<br>
[<ffffffff8108cc96>] SyS_nanosleep+0x66/0x80<br>
[<ffffffff816533d9>] system_call_fastpath+0x16/0x1b<br>
[<ffffffffffffffff>] 0xffffffffffffffff<br>
---------------------------------<br>
[<ffffffff81079271>] do_sigtimedwait+0x161/0x200<br>
[<ffffffff81079386>] SYSC_rt_sigtimedwait+0x76/0xd0<br>
[<ffffffff810793ee>] SyS_rt_sigtimedwait+0xe/0x10<br>
[<ffffffff816533d9>] system_call_fastpath+0x16/0x1b<br>
[<ffffffffffffffff>] 0xffffffffffffffff<br>
---------------------------------<br>
[<ffffffff810c277a>] futex_wait_queue_me+0xda/0x140<br>
[<ffffffff810c32be>] futex_wait+0x17e/0x290<br>
[<ffffffff810c4e26>] do_futex+0xe6/0xc30<br>
[<ffffffff810c59e1>] SyS_futex+0x71/0x150<br>
[<ffffffff816533d9>] system_call_fastpath+0x16/0x1b<br>
[<ffffffffffffffff>] 0xffffffffffffffff<br>
---------------------------------<br>
[<ffffffff810c277a>] futex_wait_queue_me+0xda/0x140<br>
[<ffffffff810c32be>] futex_wait+0x17e/0x290<br>
[<ffffffff810c4e26>] do_futex+0xe6/0xc30<br>
[<ffffffff810c59e1>] SyS_futex+0x71/0x150<br>
[<ffffffff816533d9>] system_call_fastpath+0x16/0x1b<br>
[<ffffffffffffffff>] 0xffffffffffffffff<br>
---------------------------------<br>
[<ffffffff810c277a>] futex_wait_queue_me+0xda/0x140<br>
[<ffffffff810c32be>] futex_wait+0x17e/0x290<br>
[<ffffffff810c4e26>] do_futex+0xe6/0xc30<br>
[<ffffffff810c59e1>] SyS_futex+0x71/0x150<br>
[<ffffffff816533d9>] system_call_fastpath+0x16/0x1b<br>
[<ffffffffffffffff>] 0xffffffffffffffff<br>
---------------------------------<br>
[<ffffffffa0426229>] wait_answer_interruptible+0x89/0xd0 [fuse] <<----------- This is the important thing I think<br>
[<ffffffffa0426612>] __fuse_request_send+0x232/0x290 [fuse]<br>
[<ffffffffa0426682>] fuse_request_send+0x12/0x20 [fuse]<br>
[<ffffffffa042ebea>] fuse_do_open+0xca/0x170 [fuse]<br>
[<ffffffffa042ee06>] fuse_open_common+0x56/0x80 [fuse]<br>
[<ffffffffa042ee40>] fuse_open+0x10/0x20 [fuse]<br>
[<ffffffff811a6e4b>] do_dentry_open+0x1eb/0x280<br>
[<ffffffff811a6f11>] finish_open+0x31/0x40<br>
[<ffffffff811b77ba>] do_last+0x4ca/0xe00<br>
[<ffffffff811b8510>] path_openat+0x420/0x690<br>
[<ffffffff811b8e4a>] do_filp_open+0x3a/0x90<br>
[<ffffffff811a82ee>] do_sys_open+0x12e/0x210<br>
[<ffffffff811a83ee>] SyS_open+0x1e/0x20<br>
[<ffffffff816533d9>] system_call_fastpath+0x16/0x1b<br>
[<ffffffffffffffff>] 0xffffffffffffffff<br>
---------------------------------<br>
[<ffffffff810c277a>] futex_wait_queue_me+0xda/0x140<br>
[<ffffffff810c32be>] futex_wait+0x17e/0x290<br>
[<ffffffff810c4e26>] do_futex+0xe6/0xc30<br>
[<ffffffff810c59e1>] SyS_futex+0x71/0x150<br>
[<ffffffff816533d9>] system_call_fastpath+0x16/0x1b<br>
[<ffffffffffffffff>] 0xffffffffffffffff<br>
---------------------------------<br>
[<ffffffff810c277a>] futex_wait_queue_me+0xda/0x140<br>
[<ffffffff810c32be>] futex_wait+0x17e/0x290<br>
[<ffffffff810c4e26>] do_futex+0xe6/0xc30<br>
[<ffffffff810c59e1>] SyS_futex+0x71/0x150<br>
[<ffffffff816533d9>] system_call_fastpath+0x16/0x1b<br>
[<ffffffffffffffff>] 0xffffffffffffffff<br>
---------------------------------<br>
[<ffffffff8108cb6d>] hrtimer_nanosleep+0xad/0x170<br>
[<ffffffff8108cc96>] SyS_nanosleep+0x66/0x80<br>
[<ffffffff816533d9>] system_call_fastpath+0x16/0x1b<br>
[<ffffffffffffffff>] 0xffffffffffffffff<br>
---------------------------------<br>
<br>
I don't know how to debug further but it seems like the system call hung<br></blockquote><div><br></div><div>The threads in the above process are of glusterfsd, and glusterfsd is ending up an open() attempt on a FUSE (its own) mount. Pretty obvious that it is deadlocking. Find the open()er on the symlink and you have your fix.</div>
<div><br></div><div>Avati</div></div></div></div>