<div dir="ltr"><div><div><div><div><div><div><div><div>We can use bitrot to provide a 'health' status for gluster volumes. <br>Hence I would like to propose (from a upstream/community perspective) the notion of 'health' status (as part of gluster volume info) which can derive its value from:<br></div><br>1) Bitrot<br></div> If any files are corrupted and bitrot is yet to repair them and/or its a signal to admin to do some manual operation to repair the corrupted files (for cases where we only detect, not correct)<br><br></div>2) brick status<br></div><div> Depending on brick offline/online<br></div><div><br></div>3) AFR status<br></div> Whether we have all copies in sync or not<br><br></div>This i believe is on similar lines to what Ceph does today (health status : OK, WARN, ERROR)<br></div><div>The health status derivation can be pluggable, so that in future more components can be added to query for the composite health status of the gluster volume.<br></div><div><br></div>In all of the above cases, as long as data can be served by the gluster volume reliably gluster volume status will be Started/Available, but Health status can be 'degraded' or 'warn'<br></div><div><br>This has many uses:<br><br></div><div>1) It helps provide indication to the admin that something is amiss and he can check based on:<br></div><div>bitrot / scrub status<br></div><div>brick status<br></div><div>AFR status<br><br></div><div>and take necessary action<br><br></div><div>2) It helps mgmt applns (openstack for eg) make an intelligent decision based on the health status (whether or not to pick this gluster volume for this create volume operation), so it helps acts a a coarse level filter<br><br></div><div>3) In general it gives user an idea of the health of the volume (which is different than the availability status (whether or not volume can serve data))<br></div><div>For eg: If we have a pure DHT volume, and bitrot detects silent file corruption (and we are not auto correcting) having Gluster volume status as available/started isn't entirely correct !<br></div><div><br></div><div>thanx,<br>deepak<br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 5, 2014 at 11:31 PM, Venky Shankar <span dir="ltr"><<a href="mailto:yknev.shankar@gmail.com" target="_blank">yknev.shankar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Fri, Nov 28, 2014 at 10:00 PM, Vijay Bellur <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>> wrote:<br>
> On 11/28/2014 08:30 AM, Venky Shankar wrote:<br>
>><br>
>> [snip]<br>
>>><br>
>>><br>
>>> 1. Can the bitd be one per node like self-heal-daemon and other "global"<br>
>>> services? I worry about creating 2 * N processes for N bricks in a node.<br>
>>> Maybe we can consider having one thread per volume/brick etc. in a single<br>
>>> bitd process to make it perform better.<br>
>><br>
>><br>
>> Absolutely.<br>
>> There would be one bitrot daemon per node, per volume.<br>
>><br>
><br>
> Do you foresee any problems in having one daemon per node for all volumes?<br>
<br>
</span>Not technically :). Probably that's a nice thing to do.<br>
<div><div class="h5"><br>
><br>
>><br>
>>><br>
>>> 3. I think the algorithm for checksum computation can vary within the<br>
>>> volume. I see a reference to "Hashtype is persisted along side the<br>
>>> checksum<br>
>>> and can be tuned per file type." Is this correct? If so:<br>
>>><br>
>>> a) How will the policy be exposed to the user?<br>
>><br>
>><br>
>> Bitrot daemon would have a configuration file that can be configured<br>
>> via Gluster CLI. Tuning hash types could be based on file types or<br>
>> file name patterns (regexes) [which is a bit tricky as bitrot would<br>
>> work on GFIDs rather than filenames, but this can be solved by a level<br>
>> of indirection].<br>
>><br>
>>><br>
>>> b) It would be nice to have the algorithm for computing checksums be<br>
>>> pluggable. Are there any thoughts on pluggability?<br>
>><br>
>><br>
>> Do you mean the default hash algorithm be configurable? If yes, then<br>
>> that's planned.<br>
><br>
><br>
> Sounds good.<br>
><br>
>><br>
>>><br>
>>> c) What are the steps involved in changing the hashtype/algorithm for a<br>
>>> file?<br>
>><br>
>><br>
>> Policy changes for file {types, patterns} are lazy, i.e., taken into<br>
>> effect during the next recompute. For objects that are never modified<br>
>> (after initial checksum compute), scrubbing can recompute the checksum<br>
>> using the new hash _after_ verifying the integrity of a file with the<br>
>> old hash.<br>
><br>
><br>
>><br>
>>><br>
>>> 4. Is the fop on which change detection gets triggered configurable?<br>
>><br>
>><br>
>> As of now all data modification fops trigger checksum calculation.<br>
>><br>
><br>
> Wish I was more clear on this in my OP. Is the fop on which checksum<br>
> verification/bitrot detection happens configurable? The feature page talks<br>
> about "open" being a trigger point for this. Users might want to trigger<br>
> detection on a "read" operation and not on open. It would be good to provide<br>
> this flexibility.<br>
<br>
</div></div>Ah! ok. As of now it's mostly open() and read(). Inline verification<br>
would be "off" by default due to obvious reasons.<br>
<div><div class="h5"><br>
><br>
>><br>
>>><br>
>>> 6. Any thoughts on integrating the bitrot repair framework with<br>
>>> self-heal?<br>
>><br>
>><br>
>> There are some thoughts on integration with self-heal daemon and EC.<br>
>> I'm coming up with a doc which covers those [reason for delay in<br>
>> replying to your questions ;)]. Expect the doc in in gluster-devel@<br>
>> soon.<br>
><br>
><br>
> Will look forward to this.<br>
><br>
>><br>
>>><br>
>>> 7. How does detection figure out that lazy updation is still pending and<br>
>>> not<br>
>>> raise a false positive?<br>
>><br>
>><br>
>> That's one of the things that myself and Rachana discussed yesterday.<br>
>> Should scrubbing *wait* till checksum updating is still in progress or<br>
>> is it expected that scrubbing happens when there is no active I/O<br>
>> operations on the volume (both of which imply that bitrot daemon needs<br>
>> to know when it's done it's job).<br>
>><br>
>> If both scrub and checksum updating go in parallel, then there needs<br>
>> to be way to synchronize those operations. Maybe, compute checksum on<br>
>> priority which is provided by the scrub process as a hint (that leaves<br>
>> little window for rot though) ?<br>
>><br>
>> Any thoughts?<br>
><br>
><br>
> Waiting for no active I/O in the volume might be a difficult condition to<br>
> reach in some deployments.<br>
><br>
> Some form of waiting is necessary to prevent false positives. One<br>
> possibility might be to mark an object as dirty till checksum updation is<br>
> complete. Verification/scrub can then be skipped for dirty objects.<br>
<br>
</div></div>Makes sense. Thanks!<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> -Vijay<br>
><br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br></div>