<div dir="ltr">No problem. Glad to be of help.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On 14 April 2013 05:31, Daniel Mons <span dir="ltr"><<a href="mailto:daemons@kanuka.com.au" target="_blank">daemons@kanuka.com.au</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Running the following script on an unimportant tree in the cluster<br>
this weekend as a test. So far, so good, and it appears to be doing<br>
what I want.<br>
<br>
Thanks again Pete for the recommendation.<br>
<br>
-Dan<br>
<br>
<br>
#!/bin/bash<br>
BR='-------------------------'<br>
UUID=$(/usr/bin/uuidgen)<br>
if [ "$UUID" == "" ]<br>
then<br>
echo "UUID is null"<br>
exit 1<br>
fi<br>
find "/mnt/blah/" -type f | while read FILE<br>
do<br>
DNAME=$(dirname "${FILE}")<br>
FNAME=$(basename "${FILE}")<br>
cd "${DNAME}"<br>
if (( $? > 0 ))<br>
then<br>
echo "Bad cd operation"<br>
exit 1<br>
fi<br>
pwd<br>
mv -v "${FNAME}" "${FNAME}.${UUID}"<br>
if (( $? > 0 ))<br>
then<br>
echo "Bad mv operation"<br>
exit 1<br>
fi<br>
cp -pv "${FNAME}.${UUID}" "${FNAME}"<br>
if (( $? > 0 ))<br>
then<br>
echo "Bad cp operation"<br>
exit 1<br>
fi<br>
rm -fv "${FNAME}.${UUID}"<br>
if (( $? > 0 ))<br>
then<br>
echo "Bad rm operation"<br>
exit 1<br>
fi<br>
echo "${BR}"<br>
done<br>
<br>
<br>
On 11 April 2013 22:41, Daniel Mons <<a href="mailto:daemons@kanuka.com.au">daemons@kanuka.com.au</a>> wrote:<br>
> Hi Pete,<br>
><br>
> Thanks for that link. I'm going to try this en mass on an unimportant<br>
> directory over the weekend.<br>
><br>
> -Dan<br>
><br>
><br>
> On 11 April 2013 01:41, Pete Smith <<a href="mailto:pete@realisestudio.com">pete@realisestudio.com</a>> wrote:<br>
>> Hi Dan<br>
>><br>
>> I've come up against this recently whilst trying to delete large amounts of<br>
>> files from our cluster.<br>
>><br>
>> I'm resolving it with the method from<br>
>> <a href="http://comments.gmane.org/gmane.comp.file-systems.gluster.user/1917" target="_blank">http://comments.gmane.org/gmane.comp.file-systems.gluster.user/1917</a><br>
>><br>
>> With Fabric as a helping hand, it's not too tedious.<br>
>><br>
>> Not sure about the level of glustershd compatibiity, but it's working for<br>
>> me.<br>
>><br>
>> HTH<br>
>><br>
>> Pete<br>
>> --<br>
>><br>
>><br>
>> On 10 April 2013 11:44, Daniel Mons <<a href="mailto:daemons@kanuka.com.au">daemons@kanuka.com.au</a>> wrote:<br>
>>><br>
>>> Our production GlusterFS 3.3.1GA setup is a 3x2 distribute-replicate,<br>
>>> with 100TB usable for staff. This is one of 4 identical GlusterFS<br>
>>> clusters we're running.<br>
>>><br>
>>> Very early in the life of our production Gluster rollout, we ran<br>
>>> Netatalk 2.X to share files with MacOSX clients (due to slow negative<br>
>>> lookup on CIFS/Samba for those pesky resource fork files in MacOSX's<br>
>>> Finder). Netatalk 2.X wrote it's CNID_DB files back to Gluster, which<br>
>>> caused enormous IO, locking up many nodes at a time (lots of "hung<br>
>>> task" errors in dmesg/syslog).<br>
>>><br>
>>> We've since moved to Netatalk 3.X which puts its CNID_DB files<br>
>>> elsewhere (we put them on local SSD RAID), and the lockups have<br>
>>> vanished. However, our split-brain files number in the tens of<br>
>>> thousands to to those previous lockups, and aren't always predictable<br>
>>> (i.e.: it's not always the case where brick0 is "good" and brick1 is<br>
>>> "bad"). Manually fixing the files is far too time consuming.<br>
>>><br>
>>> I've written a rudimentary script that trawls<br>
>>> /var/log/glusterfs/glustershd.log for split-brain GFIDs, tracks it<br>
>>> down on the matching pair of bricks, and figures out via a few rules<br>
>>> (size tends to be a good indicator for us, as bigger files tend to be<br>
>>> more rencent ones) which is the "good" file. This works for about 80%<br>
>>> of files, which will dramatically reduce the amount of data we have to<br>
>>> manually check.<br>
>>><br>
>>> My question is: what should I do from here? Options are:<br>
>>><br>
>>> Option 1) Delete the file from the "bad" brick<br>
>>><br>
>>> Option 2) rsync the file from the "good" brick to the "bad" brick<br>
>>> with -aX flag (preserve everything, including trusted.afr.$server and<br>
>>> trusted.gfid xattrs)<br>
>>><br>
>>> Option 3) rsync the file from "good" to "bad", and then setfattr -x<br>
>>> trusted.* on the bad brick.<br>
>>><br>
>>> Which of these is considered the better (more glustershd compatible)<br>
>>> option? Or alternatively, is there something else that's preferred?<br>
>>><br>
>>> Normally I'd just test this on our backup gluster, however as it was<br>
>>> never running Netatalk, it has no split-brain problems, so I can't<br>
>>> test the functionality.<br>
>>><br>
>>> Thanks for any insight provided,<br>
>>><br>
>>> -Dan<br>
>>> _______________________________________________<br>
>>> Gluster-users mailing list<br>
>>> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
>>> <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
>><br>
>><br>
>><br>
<span class="HOEnZb"><font color="#888888">>><br>
>> --<br>
>> Pete Smith<br>
>> DevOp/System Administrator<br>
>> Realise Studio<br>
>> 12/13 Poland Street, London W1F 8QB<br>
>> T. <a href="tel:%2B44%20%280%2920%207165%209644" value="+442071659644">+44 (0)20 7165 9644</a><br>
>><br>
>> <a href="http://realisestudio.com" target="_blank">realisestudio.com</a><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><font color="#666666"><span style="font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">Pete Smith</span><br style="padding:0px;margin:0px;font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">
<span style="font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">DevOp/System Administrator</span><br style="padding:0px;margin:0px;font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">
<span style="font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">Realise Studio</span><br style="padding:0px;margin:0px;font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">
<span style="font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">12/13 Poland Street, London W1F 8QB</span><br style="padding:0px;margin:0px;font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">
<span style="font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">T. +44 (0)20 7165 9644</span><br style="padding:0px;margin:0px;font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)">
<br style="padding:0px;margin:0px;font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)"><span style="font-family:'Lucida Console',Courier,'Courier New';font-size:12px;background-color:rgb(255,255,255)"><a href="http://realisestudio.com" target="_blank">realisestudio.com</a></span></font>
</div>