<div dir="ltr">no. I mount glusterfs<div>via fstab </div><div>do not pay any attention on the dirs structure. there were meant for other things at first :)</div><div><div>#mount volume for WIN TT 1T</div><div>stor1:HA-WIN-TT-1T /srv/nfs/HA-WIN-TT-1T glusterfs defaults,default_permissions,backupvolfile-server=stor2,direct-io-mode=enable,allow_other,max_read=131072 0 0</div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2014-10-13 19:56 GMT+03:00 Joe Julian <span dir="ltr"><<a href="mailto:joe@julianfamily.org" target="_blank">joe@julianfamily.org</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Looks like you're mounting NFS? That would be the FSCache in the
client.<div><div class="h5"><br>
<br>
<div>On 10/13/2014 09:33 AM, Roman wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">hmm,
<div>seems like another strange issue? Seen this before. Had to
restart the volume to get my empty space back.</div>
<div>
<div>root@glstor-cli:/srv/nfs/HA-WIN-TT-1T# ls -l</div>
<div>total 943718400</div>
<div>-rw-r--r-- 1 root root 966367641600 Oct 13 16:55 disk</div>
<div>root@glstor-cli:/srv/nfs/HA-WIN-TT-1T# rm disk</div>
<div>root@glstor-cli:/srv/nfs/HA-WIN-TT-1T# df -h</div>
<div>Filesystem
Size Used Avail Use% Mounted on</div>
<div>rootfs
282G 1.1G 266G 1% /</div>
<div>udev
10M 0 10M 0% /dev</div>
<div>tmpfs
1.4G 228K 1.4G 1% /run</div>
<div>/dev/disk/by-uuid/c62ee3c0-c0e5-44af-b0cd-7cb3fbcc0fba
282G 1.1G 266G 1% /</div>
<div>tmpfs
5.0M 0 5.0M 0% /run/lock</div>
<div>tmpfs
5.2G 0 5.2G 0% /run/shm</div>
<div>stor1:HA-WIN-TT-1T
1008G 901G 57G 95% /srv/nfs/HA-WIN-TT-1T</div>
</div>
<div><br>
</div>
<div>no file, but size is still 901G.</div>
<div>Both servers show the same.</div>
<div>Do I really have to restart the volume to fix that?</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2014-10-13 19:30 GMT+03:00 Roman <span dir="ltr"><<a href="mailto:romeo.r@gmail.com" target="_blank">romeo.r@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Sure.
<div>I'll let it to run for this night .</div>
</div>
<div class="gmail_extra">
<div>
<div><br>
<div class="gmail_quote">2014-10-13 19:19 GMT+03:00
Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> hi Roman,<br>
Do you think we can run this test again?
this time, could you enable 'gluster volume
profile <volname> start', do the same
test. Provide output of 'gluster volume profile
<volname> info' and logs after the test?<span><font color="#888888"><br>
<br>
Pranith</font></span>
<div>
<div><br>
<div>On 10/13/2014 09:45 PM, Roman wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Sure !
<div><br>
</div>
<div>
<div>root@stor1:~# gluster volume info</div>
<div><br>
</div>
<div>Volume Name:
HA-2TB-TT-Proxmox-cluster</div>
<div>Type: Replicate</div>
<div>Volume ID:
66e38bde-c5fa-4ce2-be6e-6b2adeaa16c2</div>
<div>Status: Started</div>
<div>Number of Bricks: 1 x 2 = 2</div>
<div>Transport-type: tcp</div>
<div>Bricks:</div>
<div>Brick1:
stor1:/exports/HA-2TB-TT-Proxmox-cluster/2TB</div>
<div>Brick2:
stor2:/exports/HA-2TB-TT-Proxmox-cluster/2TB</div>
<div>Options Reconfigured:</div>
<div>nfs.disable: 0</div>
<div>network.ping-timeout: 10</div>
<div><br>
</div>
<div>Volume Name: HA-WIN-TT-1T</div>
<div>Type: Replicate</div>
<div>Volume ID:
2937ac01-4cba-44a8-8ff8-0161b67f8ee4</div>
<div>Status: Started</div>
<div>Number of Bricks: 1 x 2 = 2</div>
<div>Transport-type: tcp</div>
<div>Bricks:</div>
<div>Brick1: stor1:/exports/NFS-WIN/1T</div>
<div>Brick2: stor2:/exports/NFS-WIN/1T</div>
<div>Options Reconfigured:</div>
<div>nfs.disable: 1</div>
<div>network.ping-timeout: 10</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2014-10-13
19:09 GMT+03:00 Pranith Kumar
Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Could you give
your 'gluster volume info' output?<br>
<br>
Pranith
<div>
<div><br>
<div>On 10/13/2014 09:36 PM,
Roman wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div>
<div dir="ltr">Hi,
<div><br>
</div>
<div>I've got this kind of
setup (servers run
replica)</div>
<div><br>
</div>
<div><br>
</div>
<div>@ 10G backend</div>
<div>gluster storage1</div>
<div>gluster storage2</div>
<div>gluster client1</div>
<div><br>
</div>
<div>@1g backend</div>
<div>other gluster clients</div>
<div><br>
</div>
<div>Servers got HW RAID5
with SAS disks.</div>
<div><br>
</div>
<div>So today I've desided
to create a 900GB file
for iscsi target that
will be located @
glusterfs separate
volume, using dd (just a
dummy file filled with
zeros, bs=1G count 900)</div>
<div>For the first of all
the process took pretty
lots of time, the
writing speed was 130
MB/sec (client port was
2 gbps, servers ports
were running @ 1gbps).</div>
<div>Then it reported
something like "endpoint
is not connected" and
all of my VMs on the
other volume started to
give me IO errors.</div>
<div>Servers load was
around 4,6 (total 12
cores)</div>
<div><br>
</div>
<div>Maybe it was due to
timeout of 2 secs, so
I've made it a big
higher, 10 sec.</div>
<div><br>
</div>
<div>Also during the dd
image creation time, VMs
very often reported me
that their disks are
slow like</div>
<div>
<p>WARNINGs: Read IO
Wait time is -0.02
(outside range [0:1]).</p>
<p>Is 130MB /sec is the
maximum bandwidth for
all of the volumes in
total? That why would
we need 10g backends?</p>
<p>HW Raid local speed
is 300 MB/sec, so it
should not be an
issue. any ideas or
mby any advices?</p>
<p><br>
</p>
<p>Maybe some1 got
optimized sysctl.conf
for 10G backend?</p>
<p>mine is pretty
simple, which can be
found from googling.</p>
<p><br>
</p>
<p>just to mention:
those VM-s were
connected using
separate 1gbps
intraface, which
means, they should not
be affected by the
client with 10g
backend.</p>
<p><br>
</p>
<p>logs are pretty
useless, they just say
this during the
outage</p>
<p><br>
</p>
<p>[2014-10-13
12:09:18.392910] W
[client-handshake.c:276:client_ping_cbk]
0-HA-2TB-TT-Proxmox-cluster-client-0:
timer must have
expired</p>
<p>[2014-10-13
12:10:08.389708] C
[client-handshake.c:127:rpc_client_ping_timer_expired]
0-HA-2TB-TT-Proxmox-cluster-client-0:
server <a href="http://10.250.0.1:49159" target="_blank">10.250.0.1:49159</a> has
not responded in the
last 2 seconds,
disconnecting.</p>
<p>[2014-10-13
12:10:08.390312] W
[client-handshake.c:276:client_ping_cbk]
0-HA-2TB-TT-Proxmox-cluster-client-0:
timer must have
expired</p>
</div>
<div>so I decided to set
the timout a bit higher.</div>
<div>
<div><br>
</div>
<div>So it seems to me,
that under high load
GlusterFS is not
useable? 130 MB/s is
not that much to get
some kind of timeouts
or makeing the systme
so slow, that VM-s
feeling themselves
bad.</div>
<div><br>
</div>
<div>Of course, after
the disconnection,
healing process was
started, but as VM-s
lost connection to
both of servers, it
was pretty useless,
they could not run
anymore. and BTW, when
u load the server with
such huge job (dd of
900GB), healing
process goes soooooo
slow :)</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
-- <br>
Best regards,<br>
Roman. </div>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Best regards,<br>
Roman. </div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
</div>
</div>
<span><font color="#888888">-- <br>
Best regards,<br>
Roman.
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Best regards,<br>
Roman.
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Best regards,<br>Roman.
</div>