<div dir="ltr">Hi Dan<div><br></div><div style>I've come up against this recently whilst trying to delete large amounts of files from our cluster.</div><div style><br></div><div style>I'm resolving it with the method from <a href="http://comments.gmane.org/gmane.comp.file-systems.gluster.user/1917">http://comments.gmane.org/gmane.comp.file-systems.gluster.user/1917</a></div>
<div style><br></div><div style>With Fabric as a helping hand, it's not too tedious.</div><div style><br></div><div style>Not sure about the level of glustershd compatibiity, but it's working for me.</div><div style>
<br></div><div style>HTH</div><div style><br></div><div style>Pete</div><div style>-- </div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 10 April 2013 11:44, Daniel Mons <span dir="ltr"><<a href="mailto:daemons@kanuka.com.au" target="_blank">daemons@kanuka.com.au</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Our production GlusterFS 3.3.1GA setup is a 3x2 distribute-replicate,<br>
with 100TB usable for staff. This is one of 4 identical GlusterFS<br>
clusters we're running.<br>
<br>
Very early in the life of our production Gluster rollout, we ran<br>
Netatalk 2.X to share files with MacOSX clients (due to slow negative<br>
lookup on CIFS/Samba for those pesky resource fork files in MacOSX's<br>
Finder). Netatalk 2.X wrote it's CNID_DB files back to Gluster, which<br>
caused enormous IO, locking up many nodes at a time (lots of "hung<br>
task" errors in dmesg/syslog).<br>
<br>
We've since moved to Netatalk 3.X which puts its CNID_DB files<br>
elsewhere (we put them on local SSD RAID), and the lockups have<br>
vanished. However, our split-brain files number in the tens of<br>
thousands to to those previous lockups, and aren't always predictable<br>
(i.e.: it's not always the case where brick0 is "good" and brick1 is<br>
"bad"). Manually fixing the files is far too time consuming.<br>
<br>
I've written a rudimentary script that trawls<br>
/var/log/glusterfs/glustershd.log for split-brain GFIDs, tracks it<br>
down on the matching pair of bricks, and figures out via a few rules<br>
(size tends to be a good indicator for us, as bigger files tend to be<br>
more rencent ones) which is the "good" file. This works for about 80%<br>
of files, which will dramatically reduce the amount of data we have to<br>
manually check.<br>
<br>
My question is: what should I do from here? Options are:<br>
<br>
Option 1) Delete the file from the "bad" brick<br>
<br>
Option 2) rsync the file from the "good" brick to the "bad" brick<br>
with -aX flag (preserve everything, including trusted.afr.$server and<br>
trusted.gfid xattrs)<br>
<br>
Option 3) rsync the file from "good" to "bad", and then setfattr -x<br>
trusted.* on the bad brick.<br>
<br>
Which of these is considered the better (more glustershd compatible)<br>
option? Or alternatively, is there something else that's preferred?<br>
<br>
Normally I'd just test this on our backup gluster, however as it was<br>
never running Netatalk, it has no split-brain problems, so I can't<br>
test the functionality.<br>
<br>
Thanks for any insight provided,<br>
<br>
-Dan<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><font color="#666666"><span style="font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">Pete Smith</span><br style="padding:0px;margin:0px;font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">
<span style="font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">DevOp/System Administrator</span><br style="padding:0px;margin:0px;font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">
<span style="font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">Realise Studio</span><br style="padding:0px;margin:0px;font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">
<span style="font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">12/13 Poland Street, London W1F 8QB</span><br style="padding:0px;margin:0px;font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">
<span style="font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">T. +44 (0)20 7165 9644</span><br style="padding:0px;margin:0px;font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">
<br style="padding:0px;margin:0px;font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)"><span style="font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)"><a href="http://realisestudio.com" target="_blank">realisestudio.com</a></span></font>
</div>