<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 10/22/2013 02:42 AM, José A. Lausuch
Sales wrote:<br>
</div>
<blockquote
cite="mid:CAL3ueginpPNbgV9FseyGiSYG4wmHrQtBfp1hBKjtOugJAgCJkw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Hi,</div>
<div><br>
</div>
<div>we are currently evaluating GlusterFS for a production
environment. Our focus is on the high-availability features of
GlusterFS. However, our tests have not worked out well. Hence
I am seeking feedback from you.</div>
<div><br>
</div>
<div><br>
</div>
<div>In our planned production environment, Gluster should
provide shared storage for VM disk images. So, our very basic
initial test setup is as follows:</div>
<div><br>
</div>
<div><br>
</div>
<div>
We are using two servers, each providing a single brick of a
replicated gluster volume (Gluster 3.4.1). A third server runs
a test-VM (Ubuntu 13.04 on QEMU 1.3.0 and libvirt 1.0.3) which
uses a disk image file stored on the gluster volume as block
device (/dev/vdb). For testing purposes, the root file system
of this VM (/dev/vda) is a disk image NOT stored on the
gluster volume.</div>
</div>
</blockquote>
<blockquote
cite="mid:CAL3ueginpPNbgV9FseyGiSYG4wmHrQtBfp1hBKjtOugJAgCJkw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div><br>
</div>
<div>To test the high-availability features of gluster under
load, we run FIO inside the VM directly on the vdb block
device (see configuration below). Up to now, we tested reading
only. The test procedure is as follows: </div>
<div><br>
</div>
<div>1.<span class="" style="white-space:pre"> </span>We start
FIO inside the VM and observe by means of "top" which of the
two servers receives the read requests (i.e., increased CPU
load of the glusterd process). Let’s say that Server1 has the
CPU load by glusterfsd.</div>
<div><br>
</div>
<div>2.<span class="" style="white-space:pre"> </span>While FIO
is running, we take down the network of this Server1 and
observe if the Server2 takes over.</div>
</div>
</blockquote>
<br>
You're bringing server1 down by taking down the NIC (assuming from
#5). This does take down the connection but it does so without
closing the TCP connection. Though this does represent worst-case
scenarios, see
<a class="moz-txt-link-freetext" href="http://joejulian.name/blog/keeping-your-vms-from-going-read-only-when-encountering-a-ping-timeout-in-glusterfs/">http://joejulian.name/blog/keeping-your-vms-from-going-read-only-when-encountering-a-ping-timeout-in-glusterfs/</a><br>
<br>
<blockquote
cite="mid:CAL3ueginpPNbgV9FseyGiSYG4wmHrQtBfp1hBKjtOugJAgCJkw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>3.<span class="" style="white-space:pre"> </span>This
“fail over” works (almost 100% of the times), we see the CPU
load from glusterfsd on Server2. As expected, Server1 does not
have any load because is “offline”.</div>
<div><br>
</div>
<div>4.<span class="" style="white-space:pre"> </span>After a
while we bring up the NIC on Server1 again. In this step we
realized that the expected behavior is that when bringing up
this NIC, this server should take over again (something like
active-passive behavior) but this happens only 5-10% of the
times. The CPU load is still on Server2.</div>
</div>
</blockquote>
I'm not sure I would have that expectation. The second server will
have taken over the open FD and the reads should come from there.
The reads for a given fd come from the first-to-respond to the
lookup().<br>
<br>
<blockquote
cite="mid:CAL3ueginpPNbgV9FseyGiSYG4wmHrQtBfp1hBKjtOugJAgCJkw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>5.<span class="" style="white-space:pre"> </span>After
some time, we bring down the NIC on Server2 expecting that
Server1 takes over. This second "fail over" crashes. The VM
complains about I/O errors which can only be resolved by
restarting the VM and sometimes even removing and creating the
volume again. <br>
</div>
</div>
</blockquote>
<blockquote
cite="mid:CAL3ueginpPNbgV9FseyGiSYG4wmHrQtBfp1hBKjtOugJAgCJkw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div><br>
</div>
<div>After some test, we realized that if restarting the
glusterd daemon (/etc/init.d/glusterd restart) on Server1
after step 3 or before step 4, the Server1 takes over
automatically without bringing down Server2 or anything like
that.</div>
</div>
</blockquote>
Check the logs for glusterd
(/var/log/glusterfs/etc-glusterfs-glusterd.vol.log) for clues.
Perhaps the <i>way</i> you're taking down the NIC is exposing some
bug. Perhaps instead of taking it down, use iptables or just killall
glusterfsd.<br>
<blockquote
cite="mid:CAL3ueginpPNbgV9FseyGiSYG4wmHrQtBfp1hBKjtOugJAgCJkw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div><br>
</div>
<div>We tested this using the normal FUSE mount and libgfapi. If
using FUSE, the local mount sometimes becomes unavailable (ls
shows not more files) if the failover fails.</div>
<div><br>
</div>
<div><br>
</div>
<div>We have a few fundamental questions in this regard:</div>
<div><br>
</div>
<div>i) Is Gluster supposed to handle such a scenario or are we
making wrong assumptions? Because the only solution we found
is to restart the daemon when a network outage occurs, but
this is not acceptable in a real scenario with VMs running
real applications.</div>
</div>
</blockquote>
I host my (raw and qcow2) vm images on a gluster volume. Since my
servers are not expected to hard-crash a lot, I take them down for
maintenance (kernel updates and such) gracefully, killing the
processes first. This closes the TCP connections and everything just
keeps humming along.<br>
<br>
<blockquote
cite="mid:CAL3ueginpPNbgV9FseyGiSYG4wmHrQtBfp1hBKjtOugJAgCJkw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>ii) What is the recommended configuration in terms of
caching (QEMU: cache=none/writethrough/writeback) and direct
I/O (FIO and Gluster) to maximize the reliability of the
failover process? We varied the parameters but could find a
working configuration. Do these parameters have an impact at
all?</div>
</div>
</blockquote>
To the best of my knowledge, none of those should affect
reliability.<br>
<br>
<blockquote
cite="mid:CAL3ueginpPNbgV9FseyGiSYG4wmHrQtBfp1hBKjtOugJAgCJkw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>FIO test specification:<br>
</div>
<div><br>
</div>
<div>[global]</div>
<div>direct=1</div>
<div>ioengine=libaio</div>
<div>iodepth=4</div>
<div>filename=/dev/vdb</div>
<div>runtime=300</div>
<div>numjobs=1</div>
<div><br>
</div>
<div>[maxthroughput]</div>
<div>rw=read</div>
<div>bs=16k</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>VM configuration:</div>
<div><br>
</div>
<div><domain type='kvm' id='6'></div>
<div> <name>testvm</name></div>
<div>
<uuid>93877c03-605b-ed67-1ab2-2ba16b5fb6b5</uuid></div>
<div> <memory unit='KiB'>2097152</memory></div>
<div> <currentMemory
unit='KiB'>2097152</currentMemory></div>
<div> <vcpu placement='static'>1</vcpu></div>
<div> <os></div>
<div> <type arch='x86_64'
machine='pc-1.1'>hvm</type></div>
<div> <boot dev='hd'/></div>
<div> </os></div>
<div> <features></div>
<div> <acpi/></div>
<div> <apic/></div>
<div> <pae/></div>
<div> </features></div>
<div> <clock offset='utc'/></div>
<div> <on_poweroff>destroy</on_poweroff></div>
<div> <on_reboot>restart</on_reboot></div>
<div> <on_crash>restart</on_crash></div>
<div> <devices></div>
<div> <emulator>/usr/bin/kvm</emulator></div>
<div> <disk type='block' device='disk'></div>
<div> <driver name='qemu' type='raw'
cache='writethrough'/></div>
<div> <source dev='/mnt/local/io-perf.img'/></div>
<div> <target dev='vda' bus='virtio'/></div>
<div> <alias name='virtio-disk0'/></div>
<div> <address type='pci' domain='0x0000' bus='0x00'
slot='0x04' function='0x0'/></div>
<div> </disk></div>
<div> <disk type='block' device='disk'></div>
<div> <driver name='qemu' type='raw'
cache='writethrough'/></div>
<div> <source
dev='/mnt/shared/io-perf-testdisk.img'/></div>
<div> <target dev='vdb' bus='virtio'/></div>
<div> <alias name='virtio-disk1'/></div>
<div> <address type='pci' domain='0x0000' bus='0x00'
slot='0x07' function='0x0'/></div>
<div> </disk></div>
<div> <controller type='usb' index='0'></div>
<div> <alias name='usb0'/></div>
<div> <address type='pci' domain='0x0000' bus='0x00'
slot='0x01' function='0x2'/></div>
<div> </controller></div>
<div> <interface type='network'></div>
<div> <mac address='52:54:00:36:5f:dd'/></div>
<div> <source network='default'/></div>
<div> <target dev='vnet0'/></div>
<div> <model type='virtio'/></div>
<div> <alias name='net0'/></div>
<div> <address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/></div>
<div> </interface></div>
<div> <input type='mouse' bus='ps2'/></div>
<div> <graphics type='vnc' port='5900' autoport='yes'
listen='127.0.0.1'></div>
<div> <listen type='address' address='127.0.0.1'/></div>
<div> </graphics></div>
<div> <video></div>
<div> <model type='cirrus' vram='9216' heads='1'/></div>
<div> <alias name='video0'/></div>
<div> <address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x0'/></div>
<div> </video></div>
<div>
<memballoon model='virtio'></div>
<div> <alias name='balloon0'/></div>
<div> <address type='pci' domain='0x0000' bus='0x00'
slot='0x05' function='0x0'/></div>
<div> </memballoon></div>
<div> </devices></div>
<div> <seclabel type='none'/></div>
<div></domain></div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Thank you very much in advance,</div>
<div>Jose Lausuch</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>