<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font color="#000099">I'm not from the glusterfs development team or
anything, but I, too started with glusterfs somewhere around the
time frame you mention, and also work with a twin-replicated setup
just like yours. <br>
<br>
When I do what you indicate here on my setup, the command
initially hangs, and on both servers for about as long as the peer
ping timeout thing (which is defaulted at 48 secs or so). After
that it works.<br>
<br>
If we can see new bugs in this setup then I would be interested,
in part because the stability of my product depends upon this,
too. Do you think you could share your glulster volume info and
gluster volume status?<br>
<br>
Also, what did heal info say before you performed this exercise?<br>
<br>
Thanks,<br>
Anirban</font><br>
<br>
<div class="moz-cite-prefix">On Sunday 23 February 2014 07:14 AM,
Greg Scott wrote:<br>
</div>
<blockquote
cite="mid:141ac001d70043f1b38101ade5606472@mail2013.infrasupport.local"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:676267712;
mso-list-type:hybrid;
mso-list-template-ids:1952608556 -615348928 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:10;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">We first went down this path back in July
2013 and now I’m back again for more. It’s a similar
situation but now with new versions of everything. I’m using
glusterfs 3.4.2 with Fedora 20.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have 2 nodes named fw1 and fw2. When I
ifdown the NIC I’m using for Gluster on either node, that node
cannot see its Gluster volume, but the other node can see it
after a timeout. As soon as I ifup that NIC, everyone can see
everything again. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Is this expected behavior? When that
interconnect drops, I want both nodes to see their own local
copy and then sync everything back up when the interconnect
connects again.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here are details. Node fw1 has an XFS
filesystem named gluster-fw1. Node fw2 has an XFS filesystem
named gluster-fw2. Those are both gluster bricks and both
nodes mount the bricks as /firewall-scripts. So anything one
node does in /firewall-scripts should also be on the other
node within a few milliseconds. The test is to isolate the
nodes from each other and see if they can still access their
own local copy of /firewall-scripts. The easiest way to do
this is to ifdown the interconnect NIC. But this doesn’t
work. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here is what happens when I ifdown the NIC
on node fw1. Node fw2 can see /firewall-scripts but fw1 shows
an error. When I ifdown on fw2, the behavior is identical,
but swapping fw1 and fw2.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">On fw1, after an ifdown I lose connection
with my Gluster filesystem.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@stylmark-fw1 firewall-scripts]#
ifdown enp5s4<o:p></o:p></p>
<p class="MsoNormal">[root@stylmark-fw1 firewall-scripts]# ls
/firewall-scripts<o:p></o:p></p>
<p class="MsoNormal">ls: cannot access /firewall-scripts:
Transport endpoint is not connected<o:p></o:p></p>
<p class="MsoNormal">[root@stylmark-fw1 firewall-scripts]# df -h<o:p></o:p></p>
<p class="MsoNormal">df: â/firewall-scriptsâ: Transport endpoint
is not connected<o:p></o:p></p>
<p class="MsoNormal">Filesystem Size Used
Avail Use% Mounted on<o:p></o:p></p>
<p class="MsoNormal">/dev/mapper/fedora-root 17G
2.2G 14G 14% /<o:p></o:p></p>
<p class="MsoNormal">devtmpfs 989M
0 989M 0% /dev<o:p></o:p></p>
<p class="MsoNormal">tmpfs 996M
0 996M 0% /dev/shm<o:p></o:p></p>
<p class="MsoNormal">tmpfs 996M
564K 996M 1% /run<o:p></o:p></p>
<p class="MsoNormal">tmpfs 996M
0 996M 0% /sys/fs/cgroup<o:p></o:p></p>
<p class="MsoNormal">tmpfs 996M
0 996M 0% /tmp<o:p></o:p></p>
<p class="MsoNormal">/dev/sda2 477M
87M 362M 20% /boot<o:p></o:p></p>
<p class="MsoNormal">/dev/sda1 200M
9.6M 191M 5% /boot/efi<o:p></o:p></p>
<p class="MsoNormal">/dev/mapper/fedora-gluster--fw1 9.8G
33M 9.8G 1% /gluster-fw1<o:p></o:p></p>
<p class="MsoNormal">10.10.10.2:/fwmaster 214G
75G 128G 37% /mnt/fwmaster<o:p></o:p></p>
<p class="MsoNormal">[root@stylmark-fw1 firewall-scripts]#<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But on fw2, I can still look at it:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@stylmark-fw2 ~]# ls /firewall-scripts<o:p></o:p></p>
<p class="MsoNormal">allow-all failover-monitor.sh
rcfirewall.conf<o:p></o:p></p>
<p class="MsoNormal">allow-all-with-nat initial_rc.firewall
start-failover-monitor.sh<o:p></o:p></p>
<p class="MsoNormal">etc rc.firewall
var<o:p></o:p></p>
<p class="MsoNormal">[root@stylmark-fw2 ~]#<o:p></o:p></p>
<p class="MsoNormal">[root@stylmark-fw2 ~]#<o:p></o:p></p>
<p class="MsoNormal">[root@stylmark-fw2 ~]# df -h<o:p></o:p></p>
<p class="MsoNormal">Filesystem Size Used
Avail Use% Mounted on<o:p></o:p></p>
<p class="MsoNormal">/dev/mapper/fedora-root 17G
2.3G 14G 14% /<o:p></o:p></p>
<p class="MsoNormal">devtmpfs 989M
0 989M 0% /dev<o:p></o:p></p>
<p class="MsoNormal">tmpfs 996M
0 996M 0% /dev/shm<o:p></o:p></p>
<p class="MsoNormal">tmpfs 996M
560K 996M 1% /run<o:p></o:p></p>
<p class="MsoNormal">tmpfs 996M
0 996M 0% /sys/fs/cgroup<o:p></o:p></p>
<p class="MsoNormal">tmpfs 996M
0 996M 0% /tmp<o:p></o:p></p>
<p class="MsoNormal">/dev/sda2 477M
87M 362M 20% /boot<o:p></o:p></p>
<p class="MsoNormal">/dev/sda1 200M
9.6M 191M 5% /boot/efi<o:p></o:p></p>
<p class="MsoNormal">/dev/mapper/fedora-gluster--fw2 9.8G
33M 9.8G 1% /gluster-fw2<o:p></o:p></p>
<p class="MsoNormal">192.168.253.2:/firewall-scripts 9.8G
33M 9.8G 1% /firewall-scripts<o:p></o:p></p>
<p class="MsoNormal">10.10.10.2:/fwmaster 214G
75G 128G 37% /mnt/fwmaster<o:p></o:p></p>
<p class="MsoNormal">[root@stylmark-fw2 ~]#<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">And back to fw1 – after an ifup, I can see
it again:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@stylmark-fw1 firewall-scripts]# ifup
enp5s4<o:p></o:p></p>
<p class="MsoNormal">[root@stylmark-fw1 firewall-scripts]#<o:p></o:p></p>
<p class="MsoNormal">[root@stylmark-fw1 firewall-scripts]# ls
/firewall-scripts<o:p></o:p></p>
<p class="MsoNormal">allow-all failover-monitor.sh
rcfirewall.conf<o:p></o:p></p>
<p class="MsoNormal">allow-all-with-nat initial_rc.firewall
start-failover-monitor.sh<o:p></o:p></p>
<p class="MsoNormal">etc rc.firewall
var<o:p></o:p></p>
<p class="MsoNormal">[root@stylmark-fw1 firewall-scripts]# df -h<o:p></o:p></p>
<p class="MsoNormal">Filesystem Size Used
Avail Use% Mounted on<o:p></o:p></p>
<p class="MsoNormal">/dev/mapper/fedora-root 17G
2.2G 14G 14% /<o:p></o:p></p>
<p class="MsoNormal">devtmpfs 989M
0 989M 0% /dev<o:p></o:p></p>
<p class="MsoNormal">tmpfs 996M
0 996M 0% /dev/shm<o:p></o:p></p>
<p class="MsoNormal">tmpfs 996M
564K 996M 1% /run<o:p></o:p></p>
<p class="MsoNormal">tmpfs 996M
0 996M 0% /sys/fs/cgroup<o:p></o:p></p>
<p class="MsoNormal">tmpfs 996M
0 996M 0% /tmp<o:p></o:p></p>
<p class="MsoNormal">/dev/sda2 477M
87M 362M 20% /boot<o:p></o:p></p>
<p class="MsoNormal">/dev/sda1 200M
9.6M 191M 5% /boot/efi<o:p></o:p></p>
<p class="MsoNormal">/dev/mapper/fedora-gluster--fw1 9.8G
33M 9.8G 1% /gluster-fw1<o:p></o:p></p>
<p class="MsoNormal">192.168.253.1:/firewall-scripts 9.8G
33M 9.8G 1% /firewall-scripts<o:p></o:p></p>
<p class="MsoNormal">10.10.10.2:/fwmaster 214G
75G 128G 37% /mnt/fwmaster<o:p></o:p></p>
<p class="MsoNormal">[root@stylmark-fw1 firewall-scripts]#<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What can I do about this?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph"
style="text-indent:-.25in;mso-list:l0 level1 lfo1"><!--[if !supportLists]--><span
style="mso-list:Ignore">-<span style="font:7.0pt "Times
New Roman"">
</span></span><!--[endif]-->Greg Scott<o:p></o:p></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>