<div dir="ltr">Hi,<br><br>I have a lab with 10 machines acting as storage servers for some compute machines, using glusterfs to distribute the data as two volumes.<br><br>Created using:<br>gluster volume create vol1 192.168.10.{221..230}:/data/vol1<br>
gluster volume create vol2 replica 2 192.168.10.{221..230}:/data/vol2<br><br>and mounted on the client and server machines using:<br>mount -t glusterfs 192.168.10.221:/vol1 /mnt/vol1<br>mount -t glusterfs 192.168.10.221:/vol2 /mnt/vol2<br>
<br>Everything worked great for almost two months now, but for some reason the bricks at 192.168.10.230 do not respond any more, making the non replicated volume very troublesome.<br><br>In the client machine under /var/log/gluster/mnt-vol1.log I see lots and lots of:<br>
0-vol1-clinet-9: remote operation failed: Transport endpoint is not connected<br>and some:<br>0-vol1-clinet-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)<br>
<br>In the server I see under /var/log/gluster/etc-glusterfs-glusterd.vol.log :<br>0-: Unknown key: brick-0<br>0-: Unknown key: brick-1<br>
0-: Unknown key: brick-2<br>
0-: Unknown key: brick-3<br>
0-: Unknown key: brick-4<br>
0-: Unknown key: brick-5<br>
0-: Unknown key: brick-6<br>
0-: Unknown key: brick-7<br>
0-: Unknown key: brick-8<br>
0-: Unknown key: brick-9<br>
...<br>0-management: setting frame-timeout to 600<br>0-management: connect returned 0<br>....<br>0-glusterd: resolve brick failed in restore<br>0-glusterd: cannot resolve brick: 192.168.10.230:/data/vol1<br>
0-glusterd: cannot resolve brick: 192.168.10.230:/data/vol2<br>
0-management: Found brick<br>....<br>0-: Stopping gluster glustershd running in pid: 3589<br>...<br>Given volfile:<br>+----------------------------------------<br>1: volume management<br>2: type mgmt/glusterd<br>3: option working directory /var/lib/glusterd<br>
4: option transport-type socket,rdma<br>...<br>8: end-volume<br>+--------------------------------------<br>0-transport: disconnecting now<br>...<br>0-management: connection to failed (Connection timed out)<br>....<br><br>
<br><br>What's the correct way to resolve this problem?<br><br>(btw - sorry I can not attach actual fragments of log - my lab is not connected to the Internet so I had to copy it manually).<br><br><br>Thanks in advance,<br>
Ayelet<br><br></div>