<br><br><div class="gmail_quote">On Wed, Aug 29, 2012 at 10:57 AM, Anand Avati <span dir="ltr"><<a href="mailto:avati@redhat.com" target="_blank">avati@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
(CC'ing gluster-devel)<br>
<br>
On 08/28/2012 09:38 PM, Raghavendra Gowdappa wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Avati,<br>
<br>
Following are the questions/thoughts related to anonymous fd framework and their usage in quick read. Please answer or give your feedback.<br>
<br>
Questions related to anonymous fd framework:<br>
==============================<u></u>==============<br>
* Anonymous fds can work because open in itself doesn't do any primary task application is interested in - like read, write etc (application does an open with an intent of doing something else). This brings in the question, why do we need open at all, can't we eliminate it altogether? If we were to eliminate open, aren't we moving from a neater to a messy design - each fop has to check whether the work associated with open (like storing contexts etc) is done in every invocation?<br>
</blockquote>
<br>
Some corrections to the above statement. There are two parts to the<br>
open() call<br>
<br>
1) The effects of the call itself. Like<br>
a) Perform permission checks and establish a 'session' (with the fd) on<br>
the allowed permission [even if permission of the inode changes in the<br>
future while the fd is still open]<br>
b) Perform additional operation like file truncation when flag O_TRUNC<br>
is specified<br>
<br>
2) Side effects of the call, like<br>
a) Specify the cache effects on future syscalls with O_[RD]SYNC,<br>
O_DIRECT flags<br>
b) Offer immunity against future calls like rename() and unlink()<br>
<br>
These are the kind of things even Gluster (or any other FS) has to<br>
guarantee with its open() syscall.<br>
<br>
Anonymous fds exist because<br>
a) Protocols like NFS3 do not support the above semantics and they are<br>
implemented completely in the client side. But we require an fd_t<br>
parameter in the read/write fops which also do not require some of the<br>
above semantics (like read/write perm checks) and other semantics are<br>
guaranteed by anonymous fds already (like immunity against rename()).<br>
Note that immunity against unlink() is currently not existing in<br>
anonymous fds.<br>
<br>
b) Internal optimizations in perf xlators do not require all the above<br>
semantics sometimes.<br>
<br>
Whether we use anonymous FDs or not, we need to keep up all the above<br>
semantics. There are some issues with the semantics even in today's<br>
version of quick-read - we assume permission check has already happened<br>
(which is usually true as FUSE performs permission checks) - but that<br>
may not be the case always. That apart, the benefit of anonymous fds in<br>
quick-read can be in handling of fd based fops in the window of time<br>
between a short-cutt'ed open() and its completion from the backend. They<br>
need not wait for the open() completion if they arrive early. Instead<br>
they can proceed with an anonymous fd -- which can significantly reduce<br>
code complexity.</blockquote><div><br> </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Again, this can be limited to O_RDONLY +<br>
~O_DIRECT|O_TRUNC flag'ed open()s</blockquote><div><br>Why is this restriction? Can you elaborate on that?<br> </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
and thereby only be vulnerable to<br>
unlink()s happening in that window.<br></blockquote><div><br>Irrespective of anonymous fds, quick read would be vulnerable to unlinks in the window bounded by open returning in application and open actually happening in backend. I am not seeing how anonymous fds alter this situation. Can you please explain?<br>
<br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
* how are ops like fsync handled with anonymous fds? How are we going to identify the fd(s) on server on which writes are actually performed? The problem is more acute if we happen to load write-behind on server side.<br>
</blockquote>
<br>
With the changes in <a href="http://review.gluster.com/712" target="_blank">http://review.gluster.com/712</a>, an fsync() fop will<br>
be a barrier against all previous writes on the inode (no matter which<br>
fd). There is no problem if you load write-behind on the server side.<br>
fsync() is essentially an inode operation and must not discriminate<br>
writes based on the fd of origin.<br></blockquote><div><br>Is this true even for fsync operation on backend filesystems? Does fsync flush changes across all fds opened on a file? <br></div><div> </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
* Though we are trying to decouple path from adressing an inode in glusterfs using nameless lookups, that decoupling is not complete. There are translators which use naming patterns to assign priorities to file (like io-cache, quick-read for the purposes of deciding whether to flush a cache or not). To be honest, the problem is seen only in fd-migration where we are using nameless lookups - for fresh lookups - in new graph, after a graph switch. Currently I am using nameless lookups with loc.path set, which solves the problem. Ideally nameless lookups are not the ones to be used during migration, since they are not meant to be used for fresh lookups (atleast till we get rid of dependencies on path based addressability internally in glusterfs). However, they have huge performance beniefits.<br>
</blockquote>
<br>
Not sure what the above point is w.r.t anonymous fds,</blockquote><div><br>Nothing related to anonymous fds themselves, but to their usage during fd migration after graph switch. After a graph switch, the first lookup in new graph is fresh one and translators like io-cache, quick-read, quota that make use of path information for their internal workings will be in trouble, if we don't have correct path in loc.path.<br>
<br><br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> but yes - nameless<br>
lookup takes away the sense of hierarchy (and "filename") and operations<br>
which depend on filename or hierarchy might not always work. But then<br>
this has been true even before we brought in nameless lookups as FUSE<br>
issues open() on an inode and therefore we are not guaranteed to perform<br>
open() on the right path when you have hardlinks.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Using anonymous fd framework in quick-read:<br>
==============================<u></u>=============<br>
* as far as quick read goes, its task becomes very simple. Just convert the fd to anonymous during open and return. It can eliminate all the dependencies of fops having to wait till open is actually done. In fact the fops it has to implement are: lookup, open and readv.<br>
</blockquote>
<br>
Look at my previous comments, it must perform a little more checks.<br>
quick-read cannot just "convert" an fd to anonymous fd. Anonymous fd has<br>
fd->pid == -1 (which a quick-unwound open() fd will not). There are also<br>
other semantics which need to be met (at least with best effort) while<br>
the actual fd is still unopened.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
* Anonymous fd awareness should be brought in afr. it shouldn't try to open the files in fops like writev if fd happens to be anonymous.<br>
</blockquote>
<br>
I think that already is the case. Also, why do you specifically mention afr?<br></blockquote><div><br>I was thinking in terms of using anonymous fds in quick-read, without having to open the file explicitly at all by delegating that responsibility (of open) to servers. Hence, I thought afr need not worry about opening the files. However, this may not work as you've explained earlier and I need to think over it.<br>
<br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Thanks,<br>
Avati<br>
<br>
<br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br>
<a href="https://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">https://lists.nongnu.org/<u></u>mailman/listinfo/gluster-devel<br clear="all"><br>-- <br>Raghavendra G<br><br>
<br><br>
</a></blockquote></div>