<html><body><div style="font-family: lucida console,sans-serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"shishir gowda" <gowda.shishir@gmail.com><br><b>To: </b>gluster-devel@nongnu.org<br><b>Sent: </b>Monday, 27 January, 2014 6:30:13 PM<br><b>Subject: </b>[Gluster-devel] bit rot support for glusterfs design draft v0.1<br><div><br></div>Hi All,<br><div><br></div>Please find the updated bit-rot design for glusterfs volumes.<br><div><br></div>Thanks to Vijay Bellur for his valuable inputs in the design.<br><div><br></div>Phase 1: File level bit rot detection<br><div><br></div>The initial approach is to achieve bit rot detection at file level,<br>where checksum is computed for a complete file, and checked during<br>access.<br><div><br></div>A single daemon(say BitD) per node will be responsible for all the<br>bricks of the node. This daemon, will be registered to the gluster<br>management daemon, and any graph changes<br>(add-brick/remove-brick/replace-brick/stop bit-rot) will be handles<br>accordingly. This BitD will register with changelog xlator of all the<br>bricks for the node, and process changes from them.</blockquote><div><br></div><div>Doesn't having a single daemon for all bricks, instead of a per brick 'bitd' introduce the potential of a performance bottleneck?<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div><br></div>Change log xlator, would give the list of files (in terms of gfid)<br>which have changed during a defined interval. Checksum's would have to<br>be computed for these based on either fd close() call for non NFS<br>access, or every write for anonymous fd access (NFS). The computed<br>checksum in addition to the timestamp of the computation would be<br>saved as a extended-attribute (xattr) of the file. By using change-log<br>xlators, we would prevent periodic scans of the bricks, to identify<br>the files whose checksums need to be updated.</blockquote><div><br></div><div>Using the changelog is a great idea, but I'd also see a requirement for an admin initiated full scan at least when bringing existing volumes under bitd control.<br></div><div><br></div><div>Also, what's the flow if the xattr is unreadable, due to bit rot. In btrfs meta data is typically mirrored.<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div><br></div>Upon access (open for non-anonymous-fd calls, every read for<br>anonymous-fd calls) from any clients, the bit rot detection xlator<br>loaded ontop of the bricks, would recompute the checksum of the file,<br>and allow the calls to proceed if they match, or fail them if they<br>mis-match. This introduces extra workload for NFS workloads, and for<br>large files which require read of the complete file to recompute the<br>checksum(we try to solve this in phase-2).</blockquote><div>every read..? That's sounds like such an overhead, admins would just turn it off.<br></div><div><br></div><div>I assume failing a read due to checksum inconsistency in a replicated volume would trigger one of the other replica's to be used, so the issue is transparent to the end user/application.<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><br><div><br></div>Since a data write happens first, followed by a delayed checksum<br>compute, there is a time frame where we might have data updated, but<br>checksums yet to be computed. We should allow the access of such files<br>if the file timestamps (mtime) has changed, and is within a defined<br>range from the current time.<br><div><br></div>Additionally, we could/should have the ability to switch of checksum<br>compute from glusterfs perspective, if the underlying FS<br>exposes/implements bit-rot detection(btrfs).</blockquote><div>+1 Why re-invent the wheel!<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div><br></div>Phase 2: Block-level(User space/defined) bit rot detection and correction.<br><div><br></div>The eventual aim is to be able to heal/correct bit rots in files. To<br>achieve this, computing checksum at a more fine grain level like a<br>block (size limited by the bit rot algorithm), so that we not only<br>detect bit rots, but also have the ability to restore them.<br>Additionally, for large files, checking the checksums at block level<br>is more efficient, rather than recompute the checksum of the whole<br>file for a an access.</blockquote><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div><br></div>In this phase, we could move the checksum computation phase to the<br>xlator loaded on-top of the posix translator at each bricks. with<br>every write, we could compute the checksum, and store the checksum and<br>continue with the write or vice versa. Every access would also be able<br>to read/compute the checksum of the requested block, check it with the<br>save checksum of the block, and act accordingly. This would take away<br>the dependency on the external BitD, and changelog xlator.<br><div><br></div>Additionally, using a Error-correcting code(ECC) or<br>Forward-error-correction(FEC) alogrithm, would enable us the correct<br>few bits in the block which have gone corrupt. And compute of the<br>complete files checksum is eliminated, as we are dealing with blocks<br>of defined size.<br><div><br></div>We require the ability to store these fine-grained checksums<br>efficiently, and extended attributes would not scale for this<br>implementation. Either a custom backed store, or a DB would be<br>preferrable in this instance.<br></blockquote><div>so if there is a per 'block' checksum, won't our capacity overheads increase to store the extra meta data, ontop of our existing replication/raid overhead?<br></div><div><br></div><div>Where does Xavi's disperse volume fit into this? Would an Erasure Coded volume lend itself easier to those use cases (cold data) where bit rot is key consideration?<br></div><div><br></div><div>If so, would a more simple bit rot strategy for gluster be<br></div><div>1) disperse volume<br></div><div>2) btrfs checksums + plumbing to trigger heal when scrub detects a problem<br></div><div><br></div><div>I like simple :)<br></div><div><br></div><div><br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">Please feel free to comment/critique.<br><div><br></div>With regards,<br>Shishir<br><div><br></div>_______________________________________________<br>Gluster-devel mailing list<br>Gluster-devel@nongnu.org<br>https://lists.nongnu.org/mailman/listinfo/gluster-devel<br></blockquote><div><br></div></div></body></html>