<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-CA link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi everyone,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>We’ve been using 3.3.2 for a while, and recently started to migrate to 3.4.2. We run on platform CentOS 6.5 for 3.4.2 (while 3.3.2 were installed on CentOS 6.4)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Recently, we’ve have a very scary condition happen and we do not know exactly the cause of it.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>We have a 3 nodes cluster with a replication factor of 3. Each node has one brick, which is made out of one RAID0 volume, comprised of multiple SSDs.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Following some read/write errors, nodes 2 and 3 have completely locked. Nothing could be done physically (nothing on the screen, nothing by SSH), physical power cycle had to be done. Node 1 was still accessible, but its fuse client rejected most if not all reads and writes.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Has anyone experienced something similar? <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Before the system freeze, the last thing the kernel seemed to be doing is killing HTTPD threads (INFO: task httpd:7910 blocked for more than 120 seconds.) End-users talk to Apache in order to read/write from the Gluster volume, so it seems a simple case of “something wrong” with gluster which locks read/writes, and eventually the kernel kills them.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>At this point, we’re unsure where to look. Nothing very specific can be found in the logs, but perhaps if someone has pointers of what to look for, that could give us a new search track.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Laurent Chouinard<o:p></o:p></p></div></body></html>