<div dir="ltr"><div>So gluster is just running on 10Mbit nic cards or 56Gbit Infiniband?<br><br><br></div><div>With 1G nic cards, assuming only replica=2, you are looking at pretty limited IO for gluster to work with. That can cause long pauses and other timeouts in my experience.<br>
</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Oct 22, 2013 at 2:42 AM, José A. Lausuch Sales <span dir="ltr"><<a href="mailto:jlausuch@gmail.com" target="_blank">jlausuch@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>we are currently evaluating GlusterFS for a production environment. Our focus is on the high-availability features of GlusterFS. However, our tests have not worked out well. Hence I am seeking feedback from you.</div>
<div><br></div><div><br></div><div>In our planned production environment, Gluster should provide shared storage for VM disk images. So, our very basic initial test setup is as follows:</div><div><br></div><div><br></div>
<div>
We are using two servers, each providing a single brick of a replicated gluster volume (Gluster 3.4.1). A third server runs a test-VM (Ubuntu 13.04 on QEMU 1.3.0 and libvirt 1.0.3) which uses a disk image file stored on the gluster volume as block device (/dev/vdb). For testing purposes, the root file system of this VM (/dev/vda) is a disk image NOT stored on the gluster volume.</div>
<div><br></div><div><br></div><div>To test the high-availability features of gluster under load, we run FIO inside the VM directly on the vdb block device (see configuration below). Up to now, we tested reading only. The test procedure is as follows: </div>
<div><br></div><div>1.<span style="white-space:pre-wrap"> </span>We start FIO inside the VM and observe by means of "top" which of the two servers receives the read requests (i.e., increased CPU load of the glusterd process). Let’s say that Server1 has the CPU load by glusterfsd.</div>
<div><br></div><div>2.<span style="white-space:pre-wrap"> </span>While FIO is running, we take down the network of this Server1 and observe if the Server2 takes over.</div><div><br></div><div>3.<span style="white-space:pre-wrap"> </span>This “fail over” works (almost 100% of the times), we see the CPU load from glusterfsd on Server2. As expected, Server1 does not have any load because is “offline”.</div>
<div><br></div><div>4.<span style="white-space:pre-wrap"> </span>After a while we bring up the NIC on Server1 again. In this step we realized that the expected behavior is that when bringing up this NIC, this server should take over again (something like active-passive behavior) but this happens only 5-10% of the times. The CPU load is still on Server2.</div>
<div><br></div><div>5.<span style="white-space:pre-wrap"> </span>After some time, we bring down the NIC on Server2 expecting that Server1 takes over. This second "fail over" crashes. The VM complains about I/O errors which can only be resolved by restarting the VM and sometimes even removing and creating the volume again. </div>
<div><br></div><div><br></div><div>After some test, we realized that if restarting the glusterd daemon (/etc/init.d/glusterd restart) on Server1 after step 3 or before step 4, the Server1 takes over automatically without bringing down Server2 or anything like that.</div>
<div><br></div><div><br></div><div>We tested this using the normal FUSE mount and libgfapi. If using FUSE, the local mount sometimes becomes unavailable (ls shows not more files) if the failover fails.</div><div><br></div>
<div><br></div><div>We have a few fundamental questions in this regard:</div><div><br></div><div>i) Is Gluster supposed to handle such a scenario or are we making wrong assumptions? Because the only solution we found is to restart the daemon when a network outage occurs, but this is not acceptable in a real scenario with VMs running real applications.</div>
<div><br></div><div>ii) What is the recommended configuration in terms of caching (QEMU: cache=none/writethrough/writeback) and direct I/O (FIO and Gluster) to maximize the reliability of the failover process? We varied the parameters but could find a working configuration. Do these parameters have an impact at all?</div>
<div><br></div><div><br></div><div><br></div><div><br></div><div>FIO test specification:<br></div><div><br></div><div>[global]</div><div>direct=1</div><div>ioengine=libaio</div><div>iodepth=4</div><div>filename=/dev/vdb</div>
<div>runtime=300</div><div>numjobs=1</div><div><br></div><div>[maxthroughput]</div><div>rw=read</div><div>bs=16k</div><div><br></div><div><br></div><div><br></div><div>VM configuration:</div><div><br></div><div><domain type='kvm' id='6'></div>
<div> <name>testvm</name></div><div> <uuid>93877c03-605b-ed67-1ab2-2ba16b5fb6b5</uuid></div><div> <memory unit='KiB'>2097152</memory></div><div> <currentMemory unit='KiB'>2097152</currentMemory></div>
<div> <vcpu placement='static'>1</vcpu></div><div> <os></div><div> <type arch='x86_64' machine='pc-1.1'>hvm</type></div><div> <boot dev='hd'/></div>
<div> </os></div><div> <features></div><div> <acpi/></div><div> <apic/></div><div> <pae/></div><div> </features></div><div> <clock offset='utc'/></div><div> <on_poweroff>destroy</on_poweroff></div>
<div> <on_reboot>restart</on_reboot></div><div> <on_crash>restart</on_crash></div><div> <devices></div><div> <emulator>/usr/bin/kvm</emulator></div><div> <disk type='block' device='disk'></div>
<div> <driver name='qemu' type='raw' cache='writethrough'/></div><div> <source dev='/mnt/local/io-perf.img'/></div><div> <target dev='vda' bus='virtio'/></div>
<div> <alias name='virtio-disk0'/></div><div> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/></div><div> </disk></div>
<div> <disk type='block' device='disk'></div><div> <driver name='qemu' type='raw' cache='writethrough'/></div><div> <source dev='/mnt/shared/io-perf-testdisk.img'/></div>
<div> <target dev='vdb' bus='virtio'/></div><div> <alias name='virtio-disk1'/></div><div> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/></div>
<div> </disk></div><div> <controller type='usb' index='0'></div><div> <alias name='usb0'/></div><div> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/></div>
<div> </controller></div><div> <interface type='network'></div><div> <mac address='52:54:00:36:5f:dd'/></div><div> <source network='default'/></div><div> <target dev='vnet0'/></div>
<div> <model type='virtio'/></div><div> <alias name='net0'/></div><div> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/></div>
<div> </interface></div><div> <input type='mouse' bus='ps2'/></div><div> <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'></div>
<div> <listen type='address' address='127.0.0.1'/></div><div> </graphics></div><div> <video></div><div> <model type='cirrus' vram='9216' heads='1'/></div>
<div> <alias name='video0'/></div><div> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/></div><div> </video></div>
<div>
<memballoon model='virtio'></div><div> <alias name='balloon0'/></div><div> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/></div>
<div> </memballoon></div><div> </devices></div><div> <seclabel type='none'/></div><div></domain></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Thank you very much in advance,</div>
<div>Jose Lausuch</div></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>