<br><br><div class="gmail_quote">On Tue Nov 25 2014 at 1:28:59 PM Shyam <<a href="mailto:srangana@redhat.com">srangana@redhat.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 11/12/2014 01:55 AM, Anand Avati wrote:<br>
><br>
><br>
> On Tue, Nov 11, 2014 at 1:56 PM, Jeff Darcy <<a href="mailto:jdarcy@redhat.com" target="_blank">jdarcy@redhat.com</a><br>
> <mailto:<a href="mailto:jdarcy@redhat.com" target="_blank">jdarcy@redhat.com</a>>> wrote:<br>
><br>
> (Personally I would have<br>
> done this by "mixing in" the parent GFID to the hash calculation, but<br>
> that alternative was ignored.)<br>
><br>
><br>
> Actually when DHT was implemented, the concept of GFID did not (yet)<br>
> exist. Due to backward compatibility it has just remained this way even<br>
> later. Including the GFID into the hash has benefits.<br>
<br>
I am curious here as this is interesting.<br>
<br>
So the layout start subvol assignment for a directory to be based on its<br>
GFID was provided so that files with the same name distribute better<br>
than ending up in the same bricks, right?<br></blockquote><div><br></div><div>Right, for e.g we wouldn't want all the README.txt in various directories of a volume to end up on the same server. The way it is achieved today is, the per server hash-range assignment is "rotated" by a certain amount (how much it is rotated is determined by a separate hash on the directory path) at the time of mkdir.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Instead as we _now_ have GFID, we could use that including the name to<br>
get a similar/better distribution, or GFID+name to determine hashed subvol.<br></blockquote><div> </div><div>What we could do now is, include the parent directory gfid as an input into the DHT hash function.</div><div><br></div><div>Today, we do approximately:</div><div> int hashval = dm_hash ("readme.txt")</div><div> hash_ranges[] = inode_ctx_get (parent_dir)</div><div> subvol = find_subvol (hash_ranges, hashval)</div><div><br></div><div>Instead, we could:</div><div> int hashval = new_hash ("readme.txt", parent_dir.gfid)</div><div> hash_ranges[] = global_value</div><div> subvol = find_subvol (hash_ranges, hashval)</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The idea here would be that on dentry creates we would need to generate<br>
the GFID and not let the bricks generate the same, so that we can choose<br>
the subvol to wind the FOP to.<br></blockquote><div><br></div><div>The GFID would be that of the parent (as an entry name is always in the context of a parent directory/inode). Also, the GFID for a new entry is already generated by the client, the brick does not generate a GFID.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
This eliminates the need for a layout per sub-directory and all the<br>
(interesting) problems that it comes with and instead can be replaced by<br>
a layout at root. Not sure if it handles all use cases and paths that we<br>
have now (which needs more understanding).<br>
<br>
I do understand there is a backward compatibility issue here, but other<br>
than this, this sounds better than the current scheme, as there is a<br>
single layout to read/optimize/stash/etc. across clients.<br>
<br>
Can I understand the rationale of this better, as to what you folks are<br>
thinking. Am I missing something or over reading on the benefits that<br>
this can provide?<br></blockquote><div><br></div><div>I think you understand it right. The benefit is one could have a single hash layout for the entire volume and the directory "specific-ness" is implemented by including the directory gfid into the hash function. The way I see it, the compromise would be something like:</div><div><br></div><div>Pro per directory range: By having per-directory hash ranges, we can do easier incremental rebalance. Partial progress is well tolerated and does not impact the entire volume. The time a given directory is undergoing rebalance, for that directory alone we need to enter "unhashed lookup" mode, only for that period of time.</div><div><br></div><div>Con per directory range: Just the new "hash assignment" phase (to impact placement of new files/data, not move old data) itself is an extended process, crawling the entire volume with complex per-directory operations. The number of points in the system where things can "break" (i.e, result in overlaps and holes in ranges) is high.</div><div><br></div><div>Pro single layout with dir GFID in hash: Avoid the numerous parts (per-dir hash ranges) which can potentially "break".</div><div><br></div><div>Con single layout with dir GFID in hash: Rebalance phase 1 (assigning new layout) is atomic for the entire volume - unhashed lookup has to be "on" for all dirs for the entire period. To mitigate this, we could explore versioning the centralized hash ranges, and store the version used by each directory in its xattrs (and update the version as the rebalance progresses). But now we have more centralized metadata (may be/ may not be a worthy compromise - not sure.)</div><div><br></div><div>In summary, including GFID into the hash calculation does open up interesting possibilities and worthy of serious consideration.</div><div><br></div><div>HTH,</div><div>Avati</div></div>