<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 08/07/2014 07:18 AM, Anand Avati
wrote:<br>
</div>
<blockquote
cite="mid:CAFboF2ydw2UYEykWmz8YRk37QnbDp8qBi7F+LgaBz9dPLjYKOA@mail.gmail.com"
type="cite">
<div dir="ltr">It would be worth checking the perf numbers without
-o acl (in case it was enabled, as seen in the other gid
thread). Client side -o acl mount option can have a negative
impact on performance because of the increased number of
up-calls from FUSE for access().</div>
</blockquote>
Actually it is all write intensive.<br>
here are the numbers they gave me from earlier runs:<br>
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop<br>
--------- ----------- ----------- -----------
------------ ----<br>
0.00 0.00 us 0.00 us 0.00 us
99 FORGET<br>
0.00 0.00 us 0.00 us 0.00 us
1093 RELEASE<br>
0.00 0.00 us 0.00 us 0.00 us 468
RELEASEDIR<br>
0.00 60.00 us 26.00 us 107.00 us
4 SETATTR<br>
0.00 91.56 us 42.00 us 157.00 us
27 UNLINK<br>
0.00 20.75 us 12.00 us 55.00 us
132 GETXATTR<br>
0.00 19.03 us 9.00 us 95.00 us
152 READLINK<br>
0.00 43.19 us 12.00 us 106.00 us
83 OPEN<br>
0.00 18.37 us 8.00 us 92.00 us
257 STATFS<br>
0.00 32.42 us 11.00 us 118.00 us
322 OPENDIR<br>
0.00 36.09 us 5.00 us 109.00 us
359 FSTAT<br>
0.00 51.14 us 37.00 us 183.00 us
663 RENAME<br>
0.00 33.32 us 6.00 us 123.00 us
1451 STAT<br>
0.00 821.79 us 21.00 us 22678.00 us
84 READ<br>
0.00 34.88 us 3.00 us 139.00 us
2326 FLUSH<br>
0.01 789.33 us 72.00 us 64054.00 us
347 CREATE<br>
0.01 1144.63 us 43.00 us 280735.00 us
337 FTRUNCATE<br>
0.01 47.82 us 16.00 us 19817.00 us
16513 LOOKUP<br>
0.02 604.85 us 11.00 us 1233.00 us
1423 READDIRP<br>
99.95 17.51 us 6.00 us 212701.00 us
300715967 WRITE<br>
<br>
Duration: 5390 seconds<br>
Data Read: 1495257497 bytes<br>
Data Written: 166546887668 bytes<br>
<br>
Pranith<br>
<blockquote
cite="mid:CAFboF2ydw2UYEykWmz8YRk37QnbDp8qBi7F+LgaBz9dPLjYKOA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<br>
</div>
<div>Thanks</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Aug 6, 2014 at 6:26 PM, Pranith
Kumar Karampuri <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class=""> <br>
<div>On 08/07/2014 06:48 AM, Anand Avati wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Aug 6, 2014 at
6:05 PM, Pranith Kumar Karampuri <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:pkarampu@redhat.com"
target="_blank">pkarampu@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">We
checked this performance with plain
distribute as well and on nfs it gave 25
minutes where as on nfs it gave around 90
minutes after disabling throttling in both
situations.</div>
</blockquote>
<div><br>
</div>
<div>This sentence is very confusing. Can you
please state it more clearly?</div>
</div>
</div>
</div>
</blockquote>
</div>
sorry :-D.<br>
We checked this performance on plain distribute volume by
disabling throttling.<br>
On nfs the run took 25 minutes.<br>
On fuse the run took 90 minutes.<span class="HOEnZb"><font
color="#888888"><br>
<br>
Pranith</font></span>
<div>
<div class="h5"><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>Thanks</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> I was
wondering if any of you guys know what
could contribute to this difference.<span><font
color="#888888"><br>
<br>
Pranith</font></span>
<div>
<div><br>
<div>On 08/07/2014 01:33 AM, Anand
Avati wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Seems like heavy
FINODELK contention. As a
diagnostic step, can you try
disabling eager-locking and check
the write performance again
(gluster volume set $name
cluster.eager-lock off)?</div>
<div class="gmail_extra"> <br>
<br>
<div class="gmail_quote">On Tue,
Aug 5, 2014 at 11:44 AM, David
F. Robinson <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:david.robinson@corvidtec.com"
target="_blank">david.robinson@corvidtec.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div>
<div>Forgot to attach
profile info in previous
email. Attached... </div>
<span><font color="#888888">
<div> </div>
<div>David</div>
</font></span>
<div>
<div>
<div> </div>
<div> </div>
<div>------ Original
Message ------</div>
<div>From: "David F.
Robinson" <<a
moz-do-not-send="true"
href="mailto:david.robinson@corvidtec.com" target="_blank">david.robinson@corvidtec.com</a>></div>
<div>To: <a
moz-do-not-send="true"
href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a></div>
<div>Sent: 8/5/2014
2:41:34 PM</div>
<div>Subject: Fw: Re:
Corvid gluster testing</div>
<div> </div>
<div>
<blockquote
cite="http://eme3fc1a31-df17-4978-a99c-f4b348e690f3@dfrobins-vaio"
type="cite">
<div>I have been
testing some of
the fixes that
Pranith incorporated
into the
3.5.2-beta to see
how they performed
for moderate
levels of i/o. All
of the stability
issues that I had
seen in previous
versions seem to
have been fixed in
3.5.2; however,
there still seem
to be some
significant
performance
issues. Pranith
suggested that I
send this to the
gluster-devel
email list, so
here goes: </div>
<div> </div>
<div>I am running an
MPI job that saves
a restart file to
the gluster file
system. When I
use the following
in my fstab to
mount the gluster
volume, the i/o
time for the 2.5GB
file is roughly
45-seconds.</div>
<div> </div>
<div><em><font> gfsib01a.corvidtec.com:/homegfs
/homegfs
glusterfs
transport=tcp,_netdev
0 0<br>
</font></em></div>
<div>When I switch
this to use the
NFS protocol (see
below), the i/o
time is
2.5-seconds.</div>
<div> </div>
<div><em><font>
gfsib01a.corvidtec.com:/homegfs
/homegfs nfs
vers=3,intr,bg,rsize=32768,wsize=32768
0 0</font></em></div>
<div> </div>
<div>The read-times
for gluster are
10-20% faster than
NFS, but the write
times are almost
20x slower. </div>
<div> </div>
<div>I am running SL
6.4 and
glusterfs-3.5.2-0.1.beta1.el6.x86_64...
</div>
<div> </div>
<div><em><font>[root@gfs01a
glusterfs]#
gluster volume
info homegfs<br>
Volume Name:
homegfs<br>
Type:
Distributed-Replicate<br>
Volume ID:
1e32672a-f1b7-4b58-ba94-58c085e59071<br>
Status:
Started<br>
Number of
Bricks: 2 x 2
= 4<br>
Transport-type:
tcp<br>
Bricks:<br>
Brick1:
gfsib01a.corvidtec.com:/data/brick01a/homegfs<br>
Brick2:
gfsib01b.corvidtec.com:/data/brick01b/homegfs<br>
Brick3:
gfsib01a.corvidtec.com:/data/brick02a/homegfs<br>
Brick4:
gfsib01b.corvidtec.com:/data/brick02b/homegfs</font></em><br>
</div>
<div> </div>
<div>David</div>
<div> </div>
<div>------
Forwarded Message
------</div>
<div>From: "Pranith
Kumar Karampuri"
<<a
moz-do-not-send="true"
href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></div>
<div>To: "David
Robinson" <<a
moz-do-not-send="true"
href="mailto:david.robinson@corvidtec.com" target="_blank">david.robinson@corvidtec.com</a>></div>
<div>Cc: "Young
Thomas" <<a
moz-do-not-send="true"
href="mailto:tom.young@corvidtec.com" target="_blank">tom.young@corvidtec.com</a>></div>
<div>Sent: 8/5/2014
2:25:38 AM</div>
<div>Subject: Re:
Corvid gluster
testing</div>
<div> </div>
<div>
<div><a
moz-do-not-send="true"
href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>
is the email-id
for the mailing
list. We should
probably start
with the initial
run numbers and
the comparison
for glusterfs
mount and nfs
mounts. May be
something like </div>
<div> </div>
<div>glusterfs
mount: 90
minutes </div>
<div>nfs mount: 25
minutes </div>
<div> </div>
<div>And profile
outputs, volume
config, number
of mounts,
hardware
configuration
should be a good
start. </div>
<div> </div>
<div>Pranith </div>
<div> </div>
<div>On 08/05/2014
09:28 AM, David
Robinson wrote:
</div>
<blockquote
type="cite">
<div>Thanks
pranith </div>
<div> </div>
<div> </div>
<div>===============================
</div>
<div>David F.
Robinson,
Ph.D. </div>
<div>President -
Corvid
Technologies </div>
<div><a
moz-do-not-send="true"
href="tel:704.799.6944%20x101" value="+17047996944" target="_blank">704.799.6944
x101</a>
[office] </div>
<div><a
moz-do-not-send="true"
href="tel:704.252.1310" value="+17042521310" target="_blank">704.252.1310</a>
[cell] </div>
<div><a
moz-do-not-send="true"
href="tel:704.799.7974" value="+17047997974" target="_blank">704.799.7974</a>
[fax] </div>
<div><a
moz-do-not-send="true"
href="mailto:David.Robinson@corvidtec.com" target="_blank">David.Robinson@corvidtec.com</a>
</div>
<div><a
moz-do-not-send="true"
href="http://www.corvidtechnologies.com/" target="_blank">http://www.corvidtechnologies.com</a>
</div>
<div> </div>
<blockquote
type="cite">
<div>On Aug 4,
2014, at 11:22
PM, Pranith
Kumar
Karampuri <<a
moz-do-not-send="true" href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>
wrote: </div>
<div> </div>
<div> </div>
<blockquote
type="cite">
<div>On
08/05/2014
08:33 AM,
Pranith Kumar
Karampuri
wrote: </div>
<div> </div>
<div>On
08/05/2014
08:29 AM,
David F.
Robinson
wrote: </div>
<blockquote
type="cite">
<blockquote
type="cite">
<blockquote
type="cite">
<div>On
08/05/2014
12:51 AM,
David F.
Robinson
wrote: </div>
<div>No. I
don't want to
use nfs. It
eliminates
most of the
benefits of
why I want to
use gluster.
Failover
redundancy of
the pair, load
balancing,
etc. </div>
</blockquote>
<div>What is
the meaning of
'Failover
redundancy of
the pair, load
balancing '
Could you
elaborate
more?
smb/nfs/glusterfs
are just
access
protocols that
gluster
supports
functionality
is almost same
</div>
</blockquote>
<div>Here is
my
understanding.
Please correct
me where I am
wrong. </div>
<div> </div>
<div>With
gluster, if I
am doing a
write and one
of the
replicated
pairs goes
down, there is
no
interruption
to the I/o.
The failover
is handled by
gluster and
the fuse
client. This
isn't done if
I use an nfs
mount unless
the component
of the pair
that goes down
isn't the one
I used for the
mount. </div>
<div> </div>
<div>With nfs,
I will have to
mount one of
the bricks.
So, if I have
gfs01a,
gfs01b,
gfs02a,
gfs02b,
gfs03a,
gfs03b, etc
and my fstab
mounts gfs01a,
it is my
understanding
that all of my
I/o will go
through gfs01a
which then
gets
distributed to
all of the
other bricks.
Gfs01a
throughput
becomes a
bottleneck.
Where if I do
a gluster
mount using
fuse, the load
balancing is
handled at the
client side ,
not the server
side. If I
have
1000-nodes
accessing
20-gluster
bricks, I need
the load
balancing
aspect. I
cannot have
all traffic
going through
the network
interface on a
single brick.
</div>
<div> </div>
<div>If I am
wrong with the
above
assumptions, I
guess my
question is
why would one
ever use the
gluster mount
instead of nfs
and/or samba?
</div>
<div> </div>
<div>Tom: feel
free to chime
in if I have
missed
anything. </div>
</blockquote>
<div>I see
your point
now. Yes the
gluster server
where you did
the mount is
kind of a
bottle neck. </div>
</blockquote>
<div>Now that
we established
the problem is
in the
clients/protocols,
you should
send out a
detailed mail
on
gluster-devel
and see if
anyone can
help with you
on performance
xlators that
can improve it
a bit more. My
area of
expertise is
more on
replication. I
am
sub-maintainer
for
replication,locks
components. I
also know
connection
management/io-threads
related issues
which lead to
hangs as I
worked on them
before.
Performance
xlators are
black box to
me. </div>
<div> </div>
<div>Performance
xlators are
enabled only
on fuse
gluster stack.
On nfs server
mounts we
disable all
the
performance
xlators except
write-behind
as nfs client
does lots of
things for
improving
performance. I
suggest you
guys follow up
more on
gluster-devel.
</div>
<div> </div>
<div>Appreciate
all the help
you did for
improving the
product :-).
Thanks a ton!
</div>
<div>Pranith </div>
<blockquote
type="cite">
<div>Pranith </div>
<blockquote
type="cite">
<div>David
(Sent from
mobile) </div>
<div> </div>
<div>===============================
</div>
<div>David F.
Robinson,
Ph.D. </div>
<div>President
- Corvid
Technologies </div>
<div><a
moz-do-not-send="true"
href="tel:704.799.6944%20x101" value="+17047996944" target="_blank">704.799.6944
x101</a>
[office] </div>
<div><a
moz-do-not-send="true"
href="tel:704.252.1310" value="+17042521310" target="_blank">704.252.1310</a>
[cell] </div>
<div><a
moz-do-not-send="true"
href="tel:704.799.7974" value="+17047997974" target="_blank">704.799.7974</a>
[fax] </div>
<div><a
moz-do-not-send="true"
href="mailto:David.Robinson@corvidtec.com" target="_blank">David.Robinson@corvidtec.com</a>
</div>
<div><a
moz-do-not-send="true"
href="http://www.corvidtechnologies.com/" target="_blank">http://www.corvidtechnologies.com</a>
</div>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<div> </div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-devel@gluster.org"
target="_blank">Gluster-devel@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel"
target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Gluster-devel mailing list
<a moz-do-not-send="true" href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>
<a moz-do-not-send="true" href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>