<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hello, <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I have had similar situation with file not listed in the ls directory listing but still available on Gluster if referred directly. It ended up a as split brain situation. After I have cleared splitbrain , files have appeared. It seems that splitbrain can occur on directory entry also, not just the file. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Here is my script to fix splitbrain on replica pairs. I am using client and server on each node. If your client is some other machine, then stat command should be executed on client host. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>WARNINIG! Script does not attempt to VALIDATE correctness of replica copy. So if you run it on replica server containing wrong copy, resulting file in the volume will be bad also. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>--------------<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>gluster volume heal VOLUME info split-brain | grep "FILENAME" | awk '{print $3}' | sort -r | uniq > /tmp/splitbrain<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>for f in $(cat /tmp/splitbrain); do<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> BRICK=/mnt/STORE_VOL/brick/<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> SBFILE="$f"<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> GFID=""<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> if [ -f ${BRICK}${SBFILE} -o -d ${BRICK}${SBFILE} ]; then<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> GFID=$(getfattr -n trusted.gfid --absolute-names -e hex ${BRICK}${SBFILE} | grep 0x | cut -d'x' -f2)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> else<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> echo "WARNING! ${BRICK}${SBFILE} not found. Skipping"<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> fi<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> if [ -f ${BRICK}/.glusterfs/${GFID:0:2}/${GFID:2:2}/${GFID:0:8}-${GFID:8:4}-${GFID:12:4}-${GFID:16:4}-${GFID:20:12} ]; then<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> echo "Removing ${BRICK}/.glusterfs/${GFID:0:2}/${GFID:2:2}/${GFID:0:8}-${GFID:8:4}-${GFID:12:4}-${GFID:16:4}-${GFID:20:12}"<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> rm -f ${BRICK}/.glusterfs/${GFID:0:2}/${GFID:2:2}/${GFID:0:8}-${GFID:8:4}-${GFID:12:4}-${GFID:16:4}-${GFID:20:12}<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> echo "Removing ${BRICK}${SBFILE}"<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> rm -f ${BRICK}${SBFILE}<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> echo "Stat'ing file"<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> stat /MOUNT_POINT/${SBFILE}<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> else<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> echo "WARNING! ${BRICK}${SBFILE} GFID file was not found. Skipping!"<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> fi<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>done<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>-------------<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> gluster-users-bounces@gluster.org [mailto:gluster-users-bounces@gluster.org] <b>On Behalf Of </b>Stefano Sinigardi<br><b>Sent:</b> 03 June 2013 13:22<br><b>To:</b> Vijay Bellur<br><b>Cc:</b> gluster-users@gluster.org; Louis Zuckerman<br><b>Subject:</b> Re: [Gluster-users] glusterfs missing files on ls<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p><div><div><div><div><div><p class=MsoNormal style='margin-left:36.0pt'>Also, because of the mistake that I did removing files and folders (the .dropbox-cache folder removal on all the bricks) directly from bricks, maybe it's better to scan the .gluster folder and remove any broken symlink before upgrading (but after having stopped glusterd, of course)?<o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt'>Please let me know. I can understand that these are basic questions, but I don't want to make many other mistakes<o:p></o:p></p></div><p class=MsoNormal style='margin-left:36.0pt'>Thanks a lot for your great support<o:p></o:p></p></div><p class=MsoNormal style='margin-left:36.0pt'>Regards,<o:p></o:p></p></div><p class=MsoNormal style='margin-left:36.0pt'> Stefano<o:p></o:p></p></div></div></body></html><html>
<head>
<title></title>
</head>
<body>
<p><br>This e-mail and any attachments are confidential and intended solely for
the addressee and may also be privileged or exempt from disclosure under
applicable law. If you are not the addressee, or have received this e-mail in
error, please notify the sender immediately, delete it from your system and do
not copy, disclose or otherwise act upon any part of this e-mail or its
attachments.<br><br>Internet communications are not guaranteed to be secure or
virus-free.<br>The Barclays Group does not accept responsibility for any loss
arising from unauthorised access to, or interference with, any Internet
communications by any third party, or from the transmission of any viruses.
Replies to this e-mail may be monitored by the Barclays Group for operational or
business reasons.<br><br>Any opinion or other information in this e-mail or its
attachments that does not relate to the business of the Barclays Group is
personal to the sender and is not given or endorsed by the Barclays
Group.<br><br>Barclays Bank PLC.Registered in England and Wales (registered no.
1026167).<br>Registered Office: 1 Churchill Place, London, E14 5HP, United
Kingdom.<br><br>Barclays Bank PLC is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority (Financial Services Register No. 122702).<br></p>
</body>
</html>