<div dir="ltr"><div class="gmail_extra">On Mon, Apr 15, 2013 at 1:19 PM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div>On 04/11/2013 05:12 AM, Brian Foster wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
On 04/10/2013 06:44 PM, Anand Avati wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
On Apr 10, 2013, at 2:50 PM, Brian Foster <<a href="mailto:bfoster@redhat.com" target="_blank">bfoster@redhat.com</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
On 04/10/2013 03:06 PM, Anand Avati wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
On Apr 10, 2013, at 6:10 AM, Jeff Darcy <<a href="mailto:jdarcy@redhat.com" target="_blank">jdarcy@redhat.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Since XFS doesn't allow hard links across directory tree quota<br>
boundaries - we get EXDEV, it would prevent gluster from creation<br>
".glusterfs" directory entries. So Gluster quota does both accounting<br>
and enforcing of quota.<br>
</blockquote>
<br>
Well, that's certainly an unfortunate consequence of using links instead<br>
of a log to track dirty files. :( One more reason to undo that damage,<br>
I suppose, but that's a different thread.<br>
</blockquote>
<br>
The problem here is with the gfid hard links (for providing stable file handles). Index uses hard links from an internal backing file which is also created within .glusterfs - the actual dirty files are never linked into index. While we do want to make index use a different format, it is completely unrelated (in fact not a problem at all) to the XFS quota issue - which is linking of true files into .glusterfs for providing stable file handles.<br>
<br>
</blockquote>
<br>
I went looking into this behavior because I don't recall running into it<br>
before. I can reproduce EXDEV if I try link from project quota A to<br>
project quota B but it looks like this only enforced when the target<br>
directory has project quota inheritance enabled (to prevent writing a<br>
file somewhere and linking it into a directory under quota, XFS enforces<br>
that the project quota IDs must match).<br>
<br>
I can create a file under a project quota and link it out of the project<br>
quota subtree without an error. So you could create a file under a<br>
project quota and link it under .glusterfs, but not vice versa and not<br>
into another directory covered by a project quota.<br>
</blockquote>
<br>
I suppose this would be a problem if we set a quota limit on the volume top level, as .glusterfs is a subdirectory of the export directory?<br>
<br>
</blockquote>
<br>
I don't think it's a severe problem. Project quotas in XFS are tracked<br>
via a project ID inode field, just like a uid or gid, only inherited via<br>
parent directory. One could conceivably set a project ID on / and<br>
recursively through every file and directory in the tree with the<br>
exception of /.glusterfs (where / refers to brick root directory).<br>
<br>
It gets a bit hairier if you want to support subquotas because you now<br>
have to emulate subquota accounting/enforcement on top of traditional<br>
project quota accounting (along with dealing with the management of<br>
project IDs between subquotas and whatnot).<br>
<br>
I'm sure there are other caveats as well, such as management of project<br>
quotas across separate volumes on the same filesystem, etc., so I'm not<br>
necessarily saying this is the way to go or not... ;)<br>
<br>
</blockquote>
<br></div></div>
I think we can consider both approaches - improving the current implementation of quotas that we have today and leveraging xfs project quotas. The former looks like something which can be implemented near term since most of the details are in place. The latter approach can be picked up once the investigations are complete and we have a more detailed understanding of the implementation details.<span><font color="#888888"><br>
<br>
-Vijay</font></span><div><div><br>
<br></div></div></blockquote>I think we should agree to get one of the quota implementation right. If we have both, it is harder to test, document and maintain the code base. <div><br></div><div>My first preference will be to leverage disk filesystem based quotas where we support at least xfs and ext4 to begin with. If we think we can do much better quota implementation entirely within the translator layer, I am supportive of it.,</div>
<div><br></div><div style>-ab</div></div>
</div></div>