<div dir="ltr"><div><div><br></div>Looks like replace-brick in a corrupted replicate brick with 3.3 doesn't work any more. To rebuild the cluster is exactly what we want to avoid when we choose to use glusterfs at the first place.<br>
<br></div>I assume to replace a failed replicate disk or node should be a standard procedure, isn't it? I could find anything related to this in the 3.3 manual.<br> <br></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Thu, Jan 10, 2013 at 12:18 PM, Daniel Taylor <span dir="ltr"><<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I've run replace-brick on missing bricks before, it should still work.<br>
<br>
On the other hand, data corruption is the worst case failure mode.<br>
The one time I hit data corruption on a node my final answer ended up being to rebuild the cluster from scratch and restore the best copy of the data I had (mix of backups and live data).<div class="im"><br>
<br>
On 01/10/2013 11:12 AM, Liang Ma wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
<br>
Thank you Daniel for you more comments.<br>
<br>
Now I can remove the damaged zfs brick after rebooting the system. But then what can I do to rejoin a new brick? I can't run gluster volume replace-brick because the old brick is gone. I can't even remove the old brick because the gluster's replicate count is 2. So what is the right procedure to replace a failed brick for replicate gluster volume?<br>
<br>
Liang<br>
<br>
<br></div><div class="im">
On Thu, Jan 10, 2013 at 11:57 AM, Daniel Taylor <<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a> <mailto:<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a>>> wrote:<br>
<br>
I'm not familiar with zfs in particular, but it should have given<br>
you a message saying why it won't unmount.<br>
<br>
In the worst case you can indeed remove the mount point from<br>
/etc/fstab and reboot. A hard reboot may be necessary in a case<br>
like this.<br>
<br>
<br>
On 01/10/2013 10:43 AM, Liang Ma wrote:<br>
<br>
<br>
Yes, I stopped the glusterfs service on the damaged system but<br>
zfs still won't allow me to umount the filesystem. Maybe I<br>
should try to shutdown the entire system.<br>
<br>
<br>
On Wed, Jan 9, 2013 at 10:28 AM, Daniel Taylor<br>
<<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a> <mailto:<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a>><br></div>
<mailto:<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a> <mailto:<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a>>><u></u>><div><div class="h5">
<br>
wrote:<br>
<br>
<br>
On 01/09/2013 08:31 AM, Liang Ma wrote:<br>
<br>
<br>
Hi Daniel,<br>
<br>
Ok, if gluster can't self-heal from this situation, I<br>
hope at<br>
least I can manually restore the volume by using the good<br>
brick available. So would you please tell me how can I<br>
"simply<br>
rebuild the filesystem and let gluster attempt to<br>
restore it<br>
from a *clean* filesystem"?<br>
<br>
<br>
Trimmed for space.<br>
<br>
You could do as Tom Pfaff suggests, but given the odds of data<br>
corruption carrying forward I'd do the following:<br>
Shut down gluster on the damaged system.<br>
Unmount the damaged filesystem.<br>
Reformat the damaged filesystem as new (throwing away any<br>
potential corruption that might not get caught on rebuild)<br>
Mount the new filesystem at the original mount point<br>
Restart gluster<br>
<br>
In the event of corruption due to hardware failure you'd<br>
be doing<br>
this on replacement hardware.<br>
The key is you have to have a functional filesystem for<br>
gluster to<br>
work with.<br>
<br>
<br>
-- Daniel Taylor VP Operations Vocal<br>
Laboratories, Inc<br>
<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a> <mailto:<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a>><br></div></div>
<mailto:<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a> <mailto:<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a>>><br>
<a href="tel:612-235-5711" value="+16122355711" target="_blank">612-235-5711</a> <tel:<a href="tel:612-235-5711" value="+16122355711" target="_blank">612-235-5711</a>><br>
<tel:<a href="tel:612-235-5711" value="+16122355711" target="_blank">612-235-5711</a> <tel:<a href="tel:612-235-5711" value="+16122355711" target="_blank">612-235-5711</a>>><div class="im"><br>
<br>
______________________________<u></u>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<u></u>org</a>><br></div>
<mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<u></u>org</a><div class="im"><br>
<mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<u></u>org</a>>><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.<u></u>org/mailman/listinfo/gluster-<u></u>users</a><br>
<br>
<br>
<br>
-- Daniel Taylor VP Operations Vocal Laboratories, Inc<br>
<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a> <mailto:<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a>> <a href="tel:612-235-5711" value="+16122355711" target="_blank">612-235-5711</a><br>
<tel:<a href="tel:612-235-5711" value="+16122355711" target="_blank">612-235-5711</a>><br>
<br>
______________________________<u></u>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<u></u>org</a>><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.<u></u>org/mailman/listinfo/gluster-<u></u>users</a><br>
<br>
<br>
</div></blockquote><div class="HOEnZb"><div class="h5">
<br>
-- <br>
Daniel Taylor VP Operations Vocal Laboratories, Inc<br>
<a href="mailto:dtaylor@vocalabs.com" target="_blank">dtaylor@vocalabs.com</a> <a href="tel:612-235-5711" value="+16122355711" target="_blank">612-235-5711</a><br>
<br>
______________________________<u></u>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.<u></u>org/mailman/listinfo/gluster-<u></u>users</a><br>
</div></div></blockquote></div><br></div>