<div dir="ltr">Hi,<div><br></div><div>I've got this kind of setup (servers run replica)</div><div><br></div><div><br></div><div>@ 10G backend</div><div>gluster storage1</div><div>gluster storage2</div><div>gluster client1</div><div><br></div><div>@1g backend</div><div>other gluster clients</div><div><br></div><div>Servers got HW RAID5 with SAS disks.</div><div><br></div><div>So today I've desided to create a 900GB file for iscsi target that will be located @ glusterfs separate volume, using dd (just a dummy file filled with zeros, bs=1G count 900)</div><div>For the first of all the process took pretty lots of time, the writing speed was 130 MB/sec (client port was 2 gbps, servers ports were running @ 1gbps).</div><div>Then it reported something like "endpoint is not connected" and all of my VMs on the other volume started to give me IO errors.</div><div>Servers load was around 4,6 (total 12 cores)</div><div><br></div><div>Maybe it was due to timeout of 2 secs, so I've made it a big higher, 10 sec.</div><div><br></div><div>Also during the dd image creation time, VMs very often reported me that their disks are slow like</div><div><p class="">WARNINGs: Read IO Wait time is -0.02 (outside range
[0:1]).</p><p class="">Is 130MB /sec is the maximum bandwidth for all of the volumes in total? That why would we need 10g backends?</p><p class="">HW Raid local speed is 300 MB/sec, so it should not be an issue. any ideas or mby any advices?</p><p class=""><br></p><p class="">Maybe some1 got optimized sysctl.conf for 10G backend?</p><p class="">mine is pretty simple, which can be found from googling.</p><p class=""><br></p><p class="">just to mention: those VM-s were connected using separate 1gbps intraface, which means, they should not be affected by the client with 10g backend.</p><p class=""><br></p><p class="">logs are pretty useless, they just say this during the outage</p><p class=""><br></p><p class="">[2014-10-13 12:09:18.392910] W [client-handshake.c:276:client_ping_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-0: timer must have expired</p><p class="">[2014-10-13 12:10:08.389708] C [client-handshake.c:127:rpc_client_ping_timer_expired] 0-HA-2TB-TT-Proxmox-cluster-client-0: server <a href="http://10.250.0.1:49159">10.250.0.1:49159</a> has not responded in the last 2 seconds, disconnecting.</p><p class="">[2014-10-13 12:10:08.390312] W [client-handshake.c:276:client_ping_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-0: timer must have expired</p></div><div>so I decided to set the timout a bit higher.</div><div><div><br></div><div>So it seems to me, that under high load GlusterFS is not useable? 130 MB/s is not that much to get some kind of timeouts or makeing the systme so slow, that VM-s feeling themselves bad.</div><div><br></div><div>Of course, after the disconnection, healing process was started, but as VM-s lost connection to both of servers, it was pretty useless, they could not run anymore. and BTW, when u load the server with such huge job (dd of 900GB), healing process goes soooooo slow :)</div><div><br></div><div><br></div><div><br></div>-- <br>Best regards,<br>Roman.
</div></div>