<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><!--ppd1000033--><div class=WordSection1><p class=MsoNormal>Looking for feedback / corrections to my logic for trying to preemptively scan GlusterFS storage for inconsistent file conditions that may prevent access from GlusterFS clients.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>This is only targeted to the 3.1.x software releases, though may be applicable to earlier and later versions.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Steps to troubleshoot client access problems (reorder when a reasonable process has been nailed down)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Check client for:<o:p></o:p></p><p class=MsoNormal> is gluster-client service running?<o:p></o:p></p><p class=MsoNormal> are GlusterFS mount points present and accessible<o:p></o:p></p><p class=MsoNormal> can other files in the same directory be accessed?<o:p></o:p></p><p class=MsoNormal> do file permissions as presented to the client prohibit client from performing whatever access is being attempted?<o:p></o:p></p><p class=MsoNormal> does lsof show the file, or the directory path to the file in use?<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Check backend storage servers for:<o:p></o:p></p><p class=MsoNormal> file presence on one pair of mirrors, and that if the file exists on the other pair of mirrors it is a GlusterFS symlink (perm 0000)<o:p></o:p></p><p class=MsoNormal> file permissions are consistent across all bricks<o:p></o:p></p><p class=MsoNormal> file attributes are consistent for all occurrences of the file (unless file is a GlusterFS symlink)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal> Logic for storage server check:<o:p></o:p></p><p class=MsoNormal>1) find all files with permissions of 0000 (GlusterFS symlink)<o:p></o:p></p><p class=MsoNormal>2) check extended attribute of each file<o:p></o:p></p><p class=MsoNormal>3) if attribute is trusted.glusterfs.dht.linkto, lookup indicated bricks and servers contained in that replica set (e.g pfs-ro1-replicate-11\000)<o:p></o:p></p><p class=MsoNormal>4) check to see that actual (normal) file exists on both of those bricks (e.g, pfs-ro1-client-22 and pfs-ro1-client-23).<o:p></o:p></p><p class=MsoNormal>5) if file does NOT exist, log to error file<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Possible auto correction steps:<o:p></o:p></p><p class=MsoNormal>1) if error file exists, process it by removing Gluster LINK files, then<o:p></o:p></p><p class=MsoNormal>2) copy error file to Gluster client node<o:p></o:p></p><p class=MsoNormal>3) on that client node, copy missing files from source to native gluster mount<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>This is the type of tool that would not only be helpful to administrators, but would increase confidence in the state of the GlusterFS storage system.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I believe that the development team is working to incorporate some of this type of functionality in newer releases of the GlusterFS software – but some of us are stuck running what we are currently running until we can make compelling arguments for the stability of newer releases.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#003D79'>James Burnash<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#003D79'>Unix Engineer<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial","sans-serif";color:#003D79'>Knight Capital Group<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div><br /><br />
<P><FONT color=navy><FONT size=1>DISCLAIMER:<BR>This e-mail, and any attachments
thereto, is intended only for use by the addressee(s)named herein and<BR>may
contain legally privileged and/or confidential information. If you are not the
intended recipient of this<BR>e-mail, you are hereby notified that any
dissemination, distribution or copying of this e-mail and any
attachments</FONT><FONT color=navy><BR></FONT><FONT size=1>thereto, is strictly
prohibited. If you have received this in error, please immediately notify me and
permanently<BR>delete the original and any printout thereof.</FONT></FONT><FONT
size=1> </FONT><FONT color=navy size=1>E-mail transmission cannot be guaranteed
to be secure or error-free.<BR>The sender therefore does not accept liability
for any errors or omissions in the contents of this message which<BR>arise as a
result of e-mail transmission.<BR>NOTICE REGARDING PRIVACY AND
CONFIDENTIALITY<BR>Knight Capital Group may, at its discretion, monitor and
review the content of all e-mail communications.</FONT></P>
<P><A href="http://www.knight.com/"><FONT color=navy
size=1>http://www.knight.com</FONT></A><BR></P>
</body></html>