<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><!--ppd1000033--><div class=WordSection1><p class=MsoNormal><span style='color:#1F497D'>Gluster 3.1.1<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>CentOS 5.5 (servers), CentOS 5.2 (client).<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>/pfs2 is the mount point for a Duplicated-Replicate volume of 4 servers.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Given this command line executed on the client:<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>root@jc1lnxsamm46:/root # time find /pfs2/online_archive/2010 -type d –print<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>and this output:<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>/pfs2/online_archive/2010/01/20<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>/pfs2/online_archive/2010/01/20/.saved<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>/pfs2/online_archive/2010/01/21<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>/pfs2/online_archive/2010/01/21/.saved<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>/pfs2/online_archive/2010/01/22<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>/pfs2/online_archive/2010/01/22/.saved<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Then no response from command for at least 8 minutes.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Client log set to debug mode, and pertinent log data available here: <a href="http://pastebin.com/6tJB5Ke8">http://pastebin.com/6tJB5Ke8</a><o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>The pieces of the log that standout look like this:<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/performance/write-beh<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> ind.so: undefined symbol: notify -- neglecting<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> [2011-04-24 12:30:08.226934] D [xlator.c:627:xlator_dynload] 0-xlator: dlsym(notify) on <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/performance/read-ahea<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> d.so: undefined symbol: notify -- neglecting<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> [2011-04-24 12:30:08.226946] D [xlator.c:644:xlator_dynload] 0-xlator: dlsym(reconfigure) on <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/performance/read<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> -ahead.so: undefined symbol: reconfigure -- neglecting<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> [2011-04-24 12:30:08.226957] D [xlator.c:650:xlator_dynload] 0-xlator: dlsym(validate_options) on <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/performance<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> /read-ahead.so: undefined symbol: validate_options -- neglecting<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> [2011-04-24 12:30:08.226999] D [xlator.c:627:xlator_dynload] 0-xlator: dlsym(notify) on <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/performance/io-cache.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> so: undefined symbol: notify -- neglecting<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:Symbol'>·</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> [2011-04-24 12:30:08.227030] D [xlator.c:627:xlator_dynload] 0-xlator: dlsym(notify) on <o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Client was originally deployed as GlusterFS 3.0.4, that was uninstalled, version 3.1.1 was installed, and then later upgraded to 3.1.3.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Any ideas on what is going on here?<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>James Burnash<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Unix Engineering<o:p></o:p></span></p></div><br /><br />
<P><FONT color=navy><FONT size=1>DISCLAIMER:<BR>This e-mail, and any attachments
thereto, is intended only for use by the addressee(s)named herein and<BR>may
contain legally privileged and/or confidential information. If you are not the
intended recipient of this<BR>e-mail, you are hereby notified that any
dissemination, distribution or copying of this e-mail and any
attachments</FONT><FONT color=navy><BR></FONT><FONT size=1>thereto, is strictly
prohibited. If you have received this in error, please immediately notify me and
permanently<BR>delete the original and any printout thereof.</FONT></FONT><FONT
size=1> </FONT><FONT color=navy size=1>E-mail transmission cannot be guaranteed
to be secure or error-free.<BR>The sender therefore does not accept liability
for any errors or omissions in the contents of this message which<BR>arise as a
result of e-mail transmission.<BR>NOTICE REGARDING PRIVACY AND
CONFIDENTIALITY<BR>Knight Capital Group may, at its discretion, monitor and
review the content of all e-mail communications.</FONT></P>
<P><A href="http://www.knight.com/"><FONT color=navy
size=1>http://www.knight.com</FONT></A><BR></P>
</body></html>