<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hello, <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>When trying to recover from failed node and replace brick with spare one I have trashed my cluster and now it is in stuck state. <o:p></o:p></p><p class=MsoNormal>Any ideas, how to reintroduce/remove those nodes and bring peace and order to cluster?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>There was a pending brick replacement operation from 0031 to 0028 (it is still not commited according to rbstate file)<o:p></o:p></p><p class=MsoNormal>There was a hardware failure on 0022 node<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I was not able to commit replace brick 0031 due to 0022 was not responding and not giving cluster lock to requesting node. <o:p></o:p></p><p class=MsoNormal>I was not able to start replacement 0022 to 0028 due to pending brick replacement<o:p></o:p></p><p class=MsoNormal>I have forced peer removal from cluster, hoping that afterwards I would be able to complete operations. Unfortunately I have removes not only 0022 but 0031 also. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I have peer probed 0031 successfully. But now gluster volume info and volume status both lists 0031 node. But when I attempt to do a brick operation I do get:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>gluster volume remove-brick glustervmstore 0031:/mnt/vmstore/brick 0036:/mnt/vmstore/brick force<o:p></o:p></p><p class=MsoNormal>Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y<o:p></o:p></p><p class=MsoNormal>Incorrect brick 0031:/mnt/vmstore/brick for volume glustervmstore<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>gluster volume replace-brick glustervmstore 0031:/mnt/vmstore/brick 0028:/mnt/vmstore/brick commit force<o:p></o:p></p><p class=MsoNormal>brick: 0031:/mnt/vmstore/brick does not exist in volume: glustervmstore<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Same applies to 0022 which is listed in volume info but not on volume status. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Full cluster volume stop and start not attempted as there are some Terabytes of information stored and actively used (Gluster used as VM storage)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thank you<o:p></o:p></p><p class=MsoNormal><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>______________________________________________________________________________<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:6.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>Elvinas Piliponis I </span><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:#7F7F7F;mso-fareast-language:EN-GB'>Virtualisation Engineering </span><span style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>Engineer</span><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'> I </span><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:#7F7F7F;mso-fareast-language:EN-GB'>Global Technology Infrastructure and Services</span><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'> </span><span lang=EN-US style='color:#1F497D;mso-fareast-language:EN-GB'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>Tel +370 5 251 1218, 7 2249 1218 I Mobile +370 6</span><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>56 69249 I Email </span><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'> </span><a href="mailto:elvinas.piliponis@barclays.com"><span style='font-size:8.0pt;font-family:"Arial","sans-serif";color:blue;mso-fareast-language:EN-GB'>elvinas.piliponis@barclays.com</span></a><span lang=EN-US style='font-size:8.0pt;color:gray;mso-fareast-language:EN-GB'><br></span><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>Barclays, GreenHall, Upes g. 21, Vilnius, Lithuania LT-081218 </span><span lang=EN-US style='color:#1F497D;mso-fareast-language:EN-GB'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>barclays.com</span><span lang=EN-US style='color:#1F497D;mso-fareast-language:EN-GB'><o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>
<html>
<head>
<title></title>
</head>
<body>
<p><br>This e-mail and any attachments are confidential and intended solely for
the addressee and may also be privileged or exempt from disclosure under
applicable law. If you are not the addressee, or have received this e-mail in
error, please notify the sender immediately, delete it from your system and do
not copy, disclose or otherwise act upon any part of this e-mail or its
attachments.<br><br>Internet communications are not guaranteed to be secure or
virus-free.<br>The Barclays Group does not accept responsibility for any loss
arising from unauthorised access to, or interference with, any Internet
communications by any third party, or from the transmission of any viruses.
Replies to this e-mail may be monitored by the Barclays Group for operational or
business reasons.<br><br>Any opinion or other information in this e-mail or its
attachments that does not relate to the business of the Barclays Group is
personal to the sender and is not given or endorsed by the Barclays
Group.<br><br>Barclays Bank PLC.Registered in England and Wales (registered no.
1026167).<br>Registered Office: 1 Churchill Place, London, E14 5HP, United
Kingdom.<br><br>Barclays Bank PLC is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority (Financial Services Register No. 122702).<br></p>
</body>
</html>