<HTML xmlns="http://www.w3.org/TR/REC-html40" xmlns:v = "urn:schemas-microsoft-com:vml" xmlns:o = "urn:schemas-microsoft-com:office:office" xmlns:w = "urn:schemas-microsoft-com:office:word" xmlns:m = "http://schemas.microsoft.com/office/2004/12/omml"><HEAD>
<META content="text/html; charset=utf-8" http-equiv=Content-Type>
<META name=Generator content="Microsoft Word 14 (filtered medium)">
<STYLE><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:676267712;
mso-list-type:hybrid;
mso-list-template-ids:1952608556 -615348928 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:10;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></STYLE>
</HEAD>
<BODY lang=EN-US link=blue vLink=purple>
<DIV>
<DIV style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri,sans-serif">Maybe I'm missing something, but if you ifconfig down the interface, you're killing the endpoint that gluster is talking to locally. It wouldn't be talking over the loop back just because its local, its talking to the IP that is now gone. I'm assuming that by downing that interface, you're also cutting off access to the subnet required to talk to the other node. Therefore, the node doesn't see either gluster endpoint. The other node has to wait the default 42 second timeout since things didn't get shut down cleanly, but then can still talk to its local gluster instances and therefore resumes.<BR><BR>I think a better way to simulate a communication failure would be to add some iptables rules to block one node from the other, or block both from seeing each other... At least then both nodes would still see the local cluster processes that are bound to the local ip address on the nics. They just wouldn't see each other.<BR><BR>A better simulation would be to down the switch port that a node is connected to, rather than the host Nic itself.<BR><BR>Todd</DIV></DIV>
<DIV dir=ltr>
<HR>
<SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri,sans-serif; FONT-WEIGHT: bold">From: </SPAN><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri,sans-serif"><A href="mailto:GregScott@infrasupport.com">Greg Scott</A></SPAN><BR><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri,sans-serif; FONT-WEIGHT: bold">Sent: </SPAN><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri,sans-serif">2/22/2014 5:44 PM</SPAN><BR><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri,sans-serif; FONT-WEIGHT: bold">To: </SPAN><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri,sans-serif"><A href="mailto:gluster-users@gluster.org">'gluster-users@gluster.org'</A></SPAN><BR><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri,sans-serif; FONT-WEIGHT: bold">Subject: </SPAN><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri,sans-serif">[Gluster-users] One node goes offline,the other node loses its connection to its local Gluster volume</SPAN><BR><BR></DIV>
<DIV class=WordSection1>
<P class=MsoNormal>We first went down this path back in July 2013 and now I’m back again for more. It’s a similar situation but now with new versions of everything. I’m using glusterfs 3.4.2 with Fedora 20. <o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>I have 2 nodes named fw1 and fw2. When I ifdown the NIC I’m using for Gluster on either node, that node cannot see its Gluster volume, but the other node can see it after a timeout. As soon as I ifup that NIC, everyone can see everything again. <o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>Is this expected behavior? When that interconnect drops, I want both nodes to see their own local copy and then sync everything back up when the interconnect connects again. <o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>Here are details. Node fw1 has an XFS filesystem named gluster-fw1. Node fw2 has an XFS filesystem named gluster-fw2. Those are both gluster bricks and both nodes mount the bricks as /firewall-scripts. So anything one node does in /firewall-scripts should also be on the other node within a few milliseconds. The test is to isolate the nodes from each other and see if they can still access their own local copy of /firewall-scripts. The easiest way to do this is to ifdown the interconnect NIC. But this doesn’t work. <o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>Here is what happens when I ifdown the NIC on node fw1. Node fw2 can see /firewall-scripts but fw1 shows an error. When I ifdown on fw2, the behavior is identical, but swapping fw1 and fw2.<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>On fw1, after an ifdown I lose connection with my Gluster filesystem.<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>[root@stylmark-fw1 firewall-scripts]# ifdown enp5s4<o:p></o:p></P>
<P class=MsoNormal>[root@stylmark-fw1 firewall-scripts]# ls /firewall-scripts<o:p></o:p></P>
<P class=MsoNormal>ls: cannot access /firewall-scripts: Transport endpoint is not connected<o:p></o:p></P>
<P class=MsoNormal>[root@stylmark-fw1 firewall-scripts]# df -h<o:p></o:p></P>
<P class=MsoNormal>df: â/firewall-scriptsâ: Transport endpoint is not connected<o:p></o:p></P>
<P class=MsoNormal>Filesystem Size Used Avail Use% Mounted on<o:p></o:p></P>
<P class=MsoNormal>/dev/mapper/fedora-root 17G 2.2G 14G 14% /<o:p></o:p></P>
<P class=MsoNormal>devtmpfs 989M 0 989M 0% /dev<o:p></o:p></P>
<P class=MsoNormal>tmpfs 996M 0 996M 0% /dev/shm<o:p></o:p></P>
<P class=MsoNormal>tmpfs 996M 564K 996M 1% /run<o:p></o:p></P>
<P class=MsoNormal>tmpfs 996M 0 996M 0% /sys/fs/cgroup<o:p></o:p></P>
<P class=MsoNormal>tmpfs 996M 0 996M 0% /tmp<o:p></o:p></P>
<P class=MsoNormal>/dev/sda2 477M 87M 362M 20% /boot<o:p></o:p></P>
<P class=MsoNormal>/dev/sda1 200M 9.6M 191M 5% /boot/efi<o:p></o:p></P>
<P class=MsoNormal>/dev/mapper/fedora-gluster--fw1 9.8G 33M 9.8G 1% /gluster-fw1<o:p></o:p></P>
<P class=MsoNormal>10.10.10.2:/fwmaster 214G 75G 128G 37% /mnt/fwmaster<o:p></o:p></P>
<P class=MsoNormal>[root@stylmark-fw1 firewall-scripts]#<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>But on fw2, I can still look at it:<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>[root@stylmark-fw2 ~]# ls /firewall-scripts<o:p></o:p></P>
<P class=MsoNormal>allow-all failover-monitor.sh rcfirewall.conf<o:p></o:p></P>
<P class=MsoNormal>allow-all-with-nat initial_rc.firewall start-failover-monitor.sh<o:p></o:p></P>
<P class=MsoNormal>etc rc.firewall var<o:p></o:p></P>
<P class=MsoNormal>[root@stylmark-fw2 ~]#<o:p></o:p></P>
<P class=MsoNormal>[root@stylmark-fw2 ~]#<o:p></o:p></P>
<P class=MsoNormal>[root@stylmark-fw2 ~]# df -h<o:p></o:p></P>
<P class=MsoNormal>Filesystem Size Used Avail Use% Mounted on<o:p></o:p></P>
<P class=MsoNormal>/dev/mapper/fedora-root 17G 2.3G 14G 14% /<o:p></o:p></P>
<P class=MsoNormal>devtmpfs 989M 0 989M 0% /dev<o:p></o:p></P>
<P class=MsoNormal>tmpfs 996M 0 996M 0% /dev/shm<o:p></o:p></P>
<P class=MsoNormal>tmpfs 996M 560K 996M 1% /run<o:p></o:p></P>
<P class=MsoNormal>tmpfs 996M 0 996M 0% /sys/fs/cgroup<o:p></o:p></P>
<P class=MsoNormal>tmpfs 996M 0 996M 0% /tmp<o:p></o:p></P>
<P class=MsoNormal>/dev/sda2 477M 87M 362M 20% /boot<o:p></o:p></P>
<P class=MsoNormal>/dev/sda1 200M 9.6M 191M 5% /boot/efi<o:p></o:p></P>
<P class=MsoNormal>/dev/mapper/fedora-gluster--fw2 9.8G 33M 9.8G 1% /gluster-fw2<o:p></o:p></P>
<P class=MsoNormal>192.168.253.2:/firewall-scripts 9.8G 33M 9.8G 1% /firewall-scripts<o:p></o:p></P>
<P class=MsoNormal>10.10.10.2:/fwmaster 214G 75G 128G 37% /mnt/fwmaster<o:p></o:p></P>
<P class=MsoNormal>[root@stylmark-fw2 ~]#<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>And back to fw1 – after an ifup, I can see it again:<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>[root@stylmark-fw1 firewall-scripts]# ifup enp5s4<o:p></o:p></P>
<P class=MsoNormal>[root@stylmark-fw1 firewall-scripts]#<o:p></o:p></P>
<P class=MsoNormal>[root@stylmark-fw1 firewall-scripts]# ls /firewall-scripts<o:p></o:p></P>
<P class=MsoNormal>allow-all failover-monitor.sh rcfirewall.conf<o:p></o:p></P>
<P class=MsoNormal>allow-all-with-nat initial_rc.firewall start-failover-monitor.sh<o:p></o:p></P>
<P class=MsoNormal>etc rc.firewall var<o:p></o:p></P>
<P class=MsoNormal>[root@stylmark-fw1 firewall-scripts]# df -h<o:p></o:p></P>
<P class=MsoNormal>Filesystem Size Used Avail Use% Mounted on<o:p></o:p></P>
<P class=MsoNormal>/dev/mapper/fedora-root 17G 2.2G 14G 14% /<o:p></o:p></P>
<P class=MsoNormal>devtmpfs 989M 0 989M 0% /dev<o:p></o:p></P>
<P class=MsoNormal>tmpfs 996M 0 996M 0% /dev/shm<o:p></o:p></P>
<P class=MsoNormal>tmpfs 996M 564K 996M 1% /run<o:p></o:p></P>
<P class=MsoNormal>tmpfs 996M 0 996M 0% /sys/fs/cgroup<o:p></o:p></P>
<P class=MsoNormal>tmpfs 996M 0 996M 0% /tmp<o:p></o:p></P>
<P class=MsoNormal>/dev/sda2 477M 87M 362M 20% /boot<o:p></o:p></P>
<P class=MsoNormal>/dev/sda1 200M 9.6M 191M 5% /boot/efi<o:p></o:p></P>
<P class=MsoNormal>/dev/mapper/fedora-gluster--fw1 9.8G 33M 9.8G 1% /gluster-fw1<o:p></o:p></P>
<P class=MsoNormal>192.168.253.1:/firewall-scripts 9.8G 33M 9.8G 1% /firewall-scripts<o:p></o:p></P>
<P class=MsoNormal>10.10.10.2:/fwmaster 214G 75G 128G 37% /mnt/fwmaster<o:p></o:p></P>
<P class=MsoNormal>[root@stylmark-fw1 firewall-scripts]#<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>What can I do about this?<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>Thanks<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoListParagraph style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><SPAN style="mso-list: Ignore">-<SPAN style="FONT: 7pt 'Times New Roman'"> </SPAN></SPAN>Greg Scott<o:p></o:p></P></DIV></BODY></HTML>