<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<span class="Apple-style-span" style="color: rgb(51, 51, 51);
font-family: 'Helvetica
Neue',Helvetica,Arial,Geneva,Verdana,sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: 16px; orphans: 2; text-align:
left; text-indent: 0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px; background-color: rgb(255, 255,
255);">I cannot stop, delete, or shrink (i.e., remove bricks from)
a GlusterFS volume. I get an "operation failed" message when I try
to do it. <br>
<br>
</span><span class="Apple-style-span" style="color: rgb(51, 51, 51);
font-family: 'Helvetica Neue', Helvetica, Arial, Geneva, Verdana,
sans-serif; font-size: 13px; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal; line-height:
16px; orphans: 2; text-align: left; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); "><span
class="Apple-style-span" style="color: rgb(51, 51, 51);
font-family: 'Helvetica Neue', Helvetica, Arial, Geneva,
Verdana, sans-serif; font-size: 13px; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: 16px; orphans: 2; text-align: left;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px;
-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); ">According to 'gluster
peer status', all peers are connected. However, I cannot add a
new peer to the pool.</span></span> <span
class="Apple-style-span" style="color: rgb(51, 51, 51);
font-family: 'Helvetica Neue', Helvetica, Arial, Geneva, Verdana,
sans-serif; font-size: 13px; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal; line-height:
16px; orphans: 2; text-align: left; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); "><span
class="Apple-style-span" style="color: rgb(51, 51, 51);
font-family: 'Helvetica Neue', Helvetica, Arial, Geneva,
Verdana, sans-serif; font-size: 13px; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: 16px; orphans: 2; text-align: left;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px;
-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); ">When I issue the
'gluster peer probe' command, the system does not issue any
errors but the 'gluster peer status' indicates that the new peer
is disconnected (all old ones are still connected). </span></span><br>
<span class="Apple-style-span" style="color: rgb(51, 51, 51);
font-family: 'Helvetica Neue', Helvetica, Arial, Geneva, Verdana,
sans-serif; font-size: 13px; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal; line-height:
16px; orphans: 2; text-align: left; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); "><br>
The only clue regarding the cause of these problems that I was
able to find are the following messages in file
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log:<br>
<br>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom:
0px; margin-left: 0px; padding-top: 0px; padding-right: 0px;
padding-bottom: 0px; padding-left: 0px; border-top-width: 0px;
border-right-width: 0px; border-bottom-width: 0px;
border-left-width: 0px; border-style: initial; border-color:
initial; outline-width: 0px; outline-style: initial;
outline-color: initial; font-size: 13px; vertical-align:
baseline; background-image: initial; background-attachment:
initial; background-origin: initial; background-clip: initial;
background-color: transparent; background-position: initial
initial; background-repeat: initial initial; ">[2011-09-28
20:50:12.128797] E
[glusterd-handler.c:1137:glusterd_handle_cli_stop_volume] 0-:
Unable to set cli op: 16
<div style="margin-top: 0px; margin-right: 0px; margin-bottom:
0px; margin-left: 0px; padding-top: 0px; padding-right: 0px;
padding-bottom: 0px; padding-left: 0px; border-top-width: 0px;
border-right-width: 0px; border-bottom-width: 0px;
border-left-width: 0px; border-style: initial; border-color:
initial; outline-width: 0px; outline-style: initial;
outline-color: initial; font-size: 13px; vertical-align:
baseline; background-image: initial; background-attachment:
initial; background-origin: initial; background-clip: initial;
background-color: transparent; background-position: initial
initial; background-repeat: initial initial; ">[2011-09-28
20:50:12.129604] W
[socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error
(Transport endpoint is not connected), peer (127.0.0.1:1008)<br>
<br>
I am getting similar messages in response to the 'gluster
volume stop/delete/remove-brick' and 'gluster peer probe'
commands. <br>
<br>
It looks like the problem is caused by the "Transport endpoint
is not connected" error for a local host but I cannot figure
out what is causing this problem. Any suggestions would be
greatly appreciated.<br>
<br>
PS. The volume is a 16-brick distributed volume with
infiniband transport (one brick per node). The OS is CentOS
5.5.<br>
<span class="Apple-style-span" style="color: rgb(51, 51, 51);
font-family: 'Helvetica Neue', Helvetica, Arial, Geneva,
Verdana, sans-serif; font-size: 13px; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: 16px; orphans: 2; text-align: left;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px;
-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); "></span><br>
</div>
</div>
</span>
<div class="moz-signature">-- <br>
Dmitry Kopelevich<br>
Associate Professor<br>
Chemical Engineering Department<br>
University of Florida<br>
Gainesville, FL 32611<br>
<br>
Phone: (352)-392-4422<br>
Fax: (352)-392-9513<br>
E-mail: <a class="moz-txt-link-abbreviated" href="mailto:dkopelevich@che.ufl.edu">dkopelevich@che.ufl.edu</a><br>
<font color="#ffffff"><span class="footer"> </span></font><br>
</div>
</body>
</html>