<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Sun, May 4, 2014 at 10:10 PM, Raghavendra G <span dir="ltr"><<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="h5">On Mon, May 5, 2014 at 12:32 AM, Anand Avati <span dir="ltr"><<a href="mailto:avati@gluster.org" target="_blank">avati@gluster.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Sun, May 4, 2014 at 9:22 AM, Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi,<br>
<br>
bug 1093324 has been opened and we have identified the following cause:<br>
<br>
1. an NFS-client does a LOOKUP of a directory on a volume<br>
2. the NFS-client receives a filehandle (contains volume-id + GFID)<br>
3. add-brick is executed, but the new brick does not have any<br>
directories yet<br>
4. the NFS-client creates a new file in the directory, this request is<br>
in the format or <filehandle>/<filename>, <filehandle> was received<br>
in step 2<br>
5. the NFS-server does a LOOKUP on the parent directory identified by<br>
the filehandle - nameless LOOKUP, only GFID is known<br>
6. the old brick(s) return successfully<br>
7. the new brick returns ESTALE<br>
8. the NFS-server returns ESTALE to the NFS-client<br>
<br>
In this case, the NFS-client should not receive an ESTALE. There is also<br>
no ESTALE error passed to the client when this procedure is done over<br>
FUSE or samba/libgfapi.<br>
<br>
Selfhealing a directory entry based only on a GFID is not always<br>
possible. Files do not have a unique filename (hardlinks), so it is not<br>
trivial to find a filename for a GFID (expensive operation, and the<br>
result could be a list). However, for a directory this is simpler.<br>
A directory is not hardlink'd in the .glusterfs directory, directories<br>
are maintained as symbolic-links. This makes it possible to find the<br>
name of a directory, when only the GFID is known.<br>
<br>
Currently DHT is not able to selfheal directories on a nameless LOOKUP.<br>
I think that it should be possible to change this, and to fix the ESTALE<br>
returned by the NFS-server.<br>
<br>
At least two changes would be needed, and this is where I would like to<br>
hear opinions from others about it:<br>
<br>
- The posix-xlator should be able to return the directory name when<br>
a GFID is given. This can be part of the LOOKUP-reply (dict), and that<br>
would add a readlink() syscall for each nameless LOOKUP that finds<br>
a directory. Or (suggested by Pranith) add a virtual xattr and handle<br>
this specific request with an additional FGETXATTR call.<br></blockquote><div><br></div></div></div><div>I think the LOOKUP-reply with readlink() is better, instead of a new over-the-wire FOP.</div><div><div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
- DHT should selfheal the directory when at least one ESTALE is returned<br>
by the bricks.</blockquote><div><br></div></div><div><br>This also makes sense, except - if even the parent directory is missing on that server (yet to be healed). Another important point to note is that, the directories (with the same GFID) themselves may be present at various locations as various dentries on the many servers. A lookup of <dir-gfid>/"name" should succeed transparently independent of the differing <dir-gfid>'s dentries across servers.</div>
</div></div></div></blockquote><div><br></div></div></div><div>Just want to be sure, among the following two scenarios<br></div><div>1. Different <pargfid>/name combinations, having same gfid<br></div><div>2. Same <pargfid>/name combination, having different gfids<br>
<br></div><div>are you saying 1 is legal (though only as a transient state during ops like rename etc)? How about 2, isn't it illegal even as a transient state (one should never ever see 2 at any point in time)?<br></div>
</div></div></div></blockquote><div><br></div><div>Rename of dirs is what causes all the interesting things. 1 is inevitable, as we just cannot rename the dir on all servers at the same time (fundamental behavior of a distributed system). 2 is something we can avoid if we make sure in our algorithms that the destination of a dir rename is empty before we start. The more important thing is the observer's point of view - if observation on server 1 is made before a rename-dir() transaction and observer on server 2 is made after, you will still see mismatching gfids etc - so it is important to be careful and not be alarmed by such false positives.</div>
<div><br></div><div><br></div></div></div></div>