<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Brian Foster" <bfoster@redhat.com><br><b>To: </b>"Shishir Gowda" <sgowda@redhat.com><br><b>Cc: </b>gluster-devel@nongnu.org<br><b>Sent: </b>Wednesday, August 7, 2013 3:11:30 AM<br><b>Subject: </b>Re: [Gluster-devel] Snapshot design for glusterfs volumes<br><div><br></div>On 08/06/2013 12:16 AM, Shishir Gowda wrote:<br>> Hi Brian, <br>> <br>> - A barrier is similar to a throttling mechanism. All it does is queue up the call_backs at the server xlator.<br>> Once barrier'ing is done, it just starts unwinding, so that clients can now get the response.<br>> The idea is that if a application does not get a acknowledgement back for the fops, it will block for sometime,<br>> hence effectively throttling itself. <br>> <br></blockquote><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">Ok, but why the need to stop unwinds? Isn't it just as effective to pend<br>the next wind from said process?<br></blockquote><div>Barrier'ing next winds face few problems:<br></div><div>1. In anonymous fd based op's, we wouldn't be able to identify the right fd. We want to barrier only the fsync call or fd's opened with O_DIRECT | O_SYNC<br></div><div>2. If writev are barrier'ed, then the buffer's also have to be saved, consuming additional space<br></div><div>3. By barrier'ing fsync response, we in effect do not allow the clients to continue write. If we ack'ed the fsync, then parallel writev might come through<br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">Maybe I missed this from the doc, but it's actually a mechanism _in_ the<br>server translator, or an independent translator? I ask because it sounds<br>like the latter could potentially provide a general throttle mechanism<br>that could be more broadly useful (over time, of course ;) ).<br></blockquote><div>We plan to introduce this change in the server xlator, as this would be the first xlator to be able to identify the fop.<br></div><div>The same could be used in the future for throttling too, as the lists would be configurable, and any fops can be added/removed.<br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">> - Snapshot here guarantees a snap of whatever has been committed onto the disk.<br>> So, in effect every internal operation (afr/dht...) should/will have to be able to heal them-selves once<br>> the volume restore takes place. <br>> <br><div><br></div>Given the explanation above to consider the barrier translator as a<br>"throttle," then I suspect its primary purpose is for performance<br>reasons as opposed to purely functional reasons (i.e., make sure the<br>snap operation occurs in a timely fashion)? My inclination when reading<br>the document was to consider the barrier mechanism as effectively the<br>quiesce portion of the typical snapshot process.<br></blockquote><div>Quiesce xlator (features/quiesce) already exists. We do not want all fops to be quiesced. That would lead to applications<br></div><div>seeing time-out, and also maintaining the queue would turn out to be a night-mare in i/o intensive work-loads<br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">From a black box perspective, it seems a little strange to me that a<br>built-in snapshot mechanism wouldn't be coherent with internal<br>operations (though from a complexity standpoint I can understand why<br>that's put off). Has there been any considerations to try and solve that<br>problem?</blockquote><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">That aside and assuming the current model, 1.) is there any assessment<br>for the likelihood of that kind of situation assuming a user follows the<br>expected process? and 2.) has the effect of that been measured on the<br>snapshot mechanism?<br></blockquote><div><div>Completly concur with you here. We have considered couple of approaches here</div><div>1. Any xlator on the server side will be made aware of a pending snap, thus giving it time for it to do its house-keeping</div><div>2. Doing the above for client side xlators is complex, and not targeted for now. But in the future if we have more control over the number/location of the clients, we should be able to handle that too</div>3. Currently most of the xlator ops are recoverable (except dht rename of directory). So at any given point in time, a snap should heal itself.</div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">It's been a while since I've played with lvm snapshots and I suppose the<br>latest technology does away with the old per-snap exception store. Even<br>still, it seems like self-heals running across sets of large,<br>inconsistent vm image files (and potentially copying them from one side<br>to another) could eat a ton of resource (space and/or cpu), no?<br></blockquote><div>This would kick in only when a snapped volume is restored, and started. In that case we have to heal.<br></div><div>Or may be alternative is, when mounted as RO, skip healing, and just make it read from the right copy.<br></div><div>One way to ease our problem here is to make sure a snap is taken only if all the bricks are up. That way self-heal might(not guaranteed) have already completed.<br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">Brian<br><div><br></div>> With regards, <br>> Shishir <br>> <br>> <hr id="zwchr"><br>> From: "Brian Foster" <bfoster@redhat.com> <br>> To: "Shishir Gowda" <sgowda@redhat.com> <br>> Cc: gluster-devel@nongnu.org <br>> Sent: Monday, August 5, 2013 6:11:47 PM <br>> Subject: Re: [Gluster-devel] Snapshot design for glusterfs volumes <br>> <br>> On 08/02/2013 02:26 AM, Shishir Gowda wrote: <br>>> Hi All, <br>>><br>>> We propose to implement snapshot support for glusterfs volumes in release-3.6. <br>>><br>>> Attaching the design document in the mail thread. <br>>><br>>> Please feel free to comment/critique. <br>>><br>> <br>> Hi Shishir, <br>> <br>> Thanks for posting this. A couple questions: <br>> <br>> - The stage-1 prepare section suggests that operations are blocked <br>> (barrier) in the callback, but later on in the doc it indicates incoming <br>> operations would be held up. Does barrier block winds and unwinds, or <br>> just winds? Could you elaborate on the logic there? <br>> <br>> - This is kind of called out in the open issues section with regard to <br>> write-behind, but don't we require some kind of operational coherency <br>> with regard to cluster translator operations? Is it expected that a <br>> snapshot across a cluster of bricks might not be coherent with regard to <br>> active afr transactions (and thus potentially require a heal in the <br>> snap), for example? <br>> <br>> Brian <br>> <br>>> With regards, <br>>> Shishir <br>>><br>>><br>>><br>>> _______________________________________________ <br>>> Gluster-devel mailing list <br>>> Gluster-devel@nongnu.org <br>>> https://lists.nongnu.org/mailman/listinfo/gluster-devel <br>>><br>> <br>> <br><div><br></div></blockquote><div><br></div></div></body></html>