<HTML><HEAD>
<STYLE id=eMClientCss>BLOCKQUOTE.cite {
PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px
}
BLOCKQUOTE.cite2 {
PADDING-TOP: 0px; PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; MARGIN-TOP: 3px; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px
}
.plain PRE {
FONT-SIZE: 100%; FONT-FAMILY: monospace; FONT-WEIGHT: normal; FONT-STYLE: normal
}
.plain TT {
FONT-SIZE: 100%; FONT-FAMILY: monospace; FONT-WEIGHT: normal; FONT-STYLE: normal
}
#b9f71ec169c34e30814de8770c02b49b {
FONT-SIZE: 11pt; FONT-FAMILY: Calibri
}
.plain PRE {
FONT-SIZE: 11pt; FONT-FAMILY: Calibri
}
.plain TT {
FONT-SIZE: 11pt; FONT-FAMILY: Calibri
}
BODY {
FONT-SIZE: 11pt; FONT-FAMILY: Calibri
}
</STYLE>
<STYLE></STYLE>
</HEAD>
<BODY scroll=auto bgColor=#ffffff class>
<DIV>Just to clarify a little, there are two cases where I was evaluating performance.</DIV>
<DIV> </DIV>
<DIV>1) The first case that Pranith was working involved 20-nodes with 4-processors on each node for a total of 80-processors. Each processor does its own independent i/o. These files are roughly 100-200MB each and there are several hundred of them. When I mounted the gluster system using fuse, it took 1.5-hours to do the i/o. When I mounted the same system using NFS, it took 30-minutes. Note, that in order to get the gluster mounted file-system down to 1.5-hours, I had to get rid of the replicated volume (this was done during troubleshooting with Pranith to rule out other possible issues). The timing was significantly worse (3+ hours) when I was using a replicated pair.</DIV>
<DIV>2) The second case was the output of a larger single file (roughly 2.5TB). For this case, it takes the gluster mounted filesystem 60-seconds (although I got that down to 52-seconds with some gluster parameter tuning). The NFS mount takes 38-seconds. I sent the results of this to the developer list first as this case is much easier to test (50-seconds versus what could be 3+ hours).</DIV>
<DIV> </DIV>
<DIV>I am head out of town for a few days and will not be able to do additional testing until Monday. For the second case, I will turn off <SPAN id=b9f71ec169c34e30814de8770c02b49b style="BACKGROUND-COLOR: #ffffff">cluster.eager-lock and send the results to the email list. If there is any other testing that you would like to see for the first case, let me know and I will be happy to perform the tests and send in the results... </SPAN></DIV>
<DIV><SPAN style="BACKGROUND-COLOR: #ffffff"></SPAN> </DIV>
<DIV><SPAN style="BACKGROUND-COLOR: #ffffff">Sorry for the confusion... </SPAN></DIV>
<DIV><SPAN style="BACKGROUND-COLOR: #ffffff"></SPAN> </DIV>
<DIV>David</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>------ Original Message ------</DIV>
<DIV>From: "Pranith Kumar Karampuri" <<A href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</A>></DIV>
<DIV>To: "Anand Avati" <<A href="mailto:avati@gluster.org">avati@gluster.org</A>></DIV>
<DIV>Cc: "David F. Robinson" <<A href="mailto:david.robinson@corvidtec.com">david.robinson@corvidtec.com</A>>; "Gluster Devel" <<A href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</A>></DIV>
<DIV>Sent: 8/6/2014 9:51:11 PM</DIV>
<DIV>Subject: Re: [Gluster-devel] Fw: Re: Corvid gluster testing</DIV>
<DIV> </DIV>
<DIV id=afbc941e1b294c27ae11325ee3d45b43 style="COLOR: #000000">
<BLOCKQUOTE class=cite2 cite=53E2DB8F.50604@redhat.com type="cite"><BR>
<DIV class=moz-cite-prefix>On 08/07/2014 07:18 AM, Anand Avati wrote:<BR></DIV>
<BLOCKQUOTE class=cite cite=mid:CAFboF2ydw2UYEykWmz8YRk37QnbDp8qBi7F+LgaBz9dPLjYKOA@mail.gmail.com type="cite">
<DIV dir=ltr>It would be worth checking the perf numbers without -o acl (in case it was enabled, as seen in the other gid thread). Client side -o acl mount option can have a negative impact on performance because of the increased number of up-calls from FUSE for access().</DIV></BLOCKQUOTE>Actually it is all write intensive.<BR>here are the numbers they gave me from earlier runs:<BR> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<BR> --------- ----------- ----------- ----------- ------------ ----<BR> 0.00 0.00 us 0.00 us 0.00 us 99 FORGET<BR> 0.00 0.00 us 0.00 us 0.00 us 1093 RELEASE<BR> 0.00 0.00 us 0.00 us 0.00 us 468 RELEASEDIR<BR> 0.00 60.00 us 26.00 us 107.00 us 4 SETATTR<BR> 0.00 91.56 us 42.00 us 157.00 us 27 UNLINK<BR> 0.00 20.75 us 12.00 us 55.00 us 132 GETXATTR<BR> 0.00 19.03 us 9.00 us 95.00 us 152 READLINK<BR> 0.00 43.19 us 12.00 us 106.00 us 83 OPEN<BR> 0.00 18.37 us 8.00 us 92.00 us 257 STATFS<BR> 0.00 32.42 us 11.00 us 118.00 us 322 OPENDIR<BR> 0.00 36.09 us 5.00 us 109.00 us 359 FSTAT<BR> 0.00 51.14 us 37.00 us 183.00 us 663 RENAME<BR> 0.00 33.32 us 6.00 us 123.00 us 1451 STAT<BR> 0.00 821.79 us 21.00 us 22678.00 us 84 READ<BR> 0.00 34.88 us 3.00 us 139.00 us 2326 FLUSH<BR> 0.01 789.33 us 72.00 us 64054.00 us 347 CREATE<BR> 0.01 1144.63 us 43.00 us 280735.00 us 337 FTRUNCATE<BR> 0.01 47.82 us 16.00 us 19817.00 us 16513 LOOKUP<BR> 0.02 604.85 us 11.00 us 1233.00 us 1423 READDIRP<BR> 99.95 17.51 us 6.00 us 212701.00 us 300715967 WRITE<BR> <BR> Duration: 5390 seconds<BR> Data Read: 1495257497 bytes<BR>Data Written: 166546887668 bytes<BR><BR>Pranith<BR>
<BLOCKQUOTE class=cite cite=mid:CAFboF2ydw2UYEykWmz8YRk37QnbDp8qBi7F+LgaBz9dPLjYKOA@mail.gmail.com type="cite">
<DIV dir=ltr>
<DIV><BR></DIV>
<DIV>Thanks</DIV></DIV>
<DIV class=gmail_extra><BR><BR>
<DIV class=gmail_quote>On Wed, Aug 6, 2014 at 6:26 PM, Pranith Kumar Karampuri <SPAN dir=ltr><<A href="mailto:pkarampu@redhat.com" moz-do-not-send="true">pkarampu@redhat.com</A>></SPAN> wrote:<BR>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<DIV bgcolor="#FFFFFF" text="#000000">
<DIV><BR>
<DIV>On 08/07/2014 06:48 AM, Anand Avati wrote:<BR></DIV>
<BLOCKQUOTE class=cite type="cite">
<DIV dir=ltr><BR>
<DIV class=gmail_extra><BR><BR>
<DIV class=gmail_quote>On Wed, Aug 6, 2014 at 6:05 PM, Pranith Kumar Karampuri <SPAN dir=ltr><<A href="mailto:pkarampu@redhat.com" moz-do-not-send="true">pkarampu@redhat.com</A>></SPAN> wrote:<BR>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<DIV bgcolor="#FFFFFF" text="#000000">We checked this performance with plain distribute as well and on nfs it gave 25 minutes where as on nfs it gave around 90 minutes after disabling throttling in both situations.</DIV></BLOCKQUOTE>
<DIV><BR></DIV>
<DIV>This sentence is very confusing. Can you please state it more clearly?</DIV></DIV></DIV></DIV></BLOCKQUOTE></DIV>sorry :-D.<BR>We checked this performance on plain distribute volume by disabling throttling.<BR>On nfs the run took 25 minutes.<BR>On fuse the run took 90 minutes.<SPAN class=HOEnZb><FONT color=#888888><BR><BR>Pranith</FONT></SPAN>
<DIV>
<DIV class=h5><BR>
<BLOCKQUOTE class=cite type="cite">
<DIV dir=ltr>
<DIV class=gmail_extra>
<DIV class=gmail_quote>
<DIV><BR></DIV>
<DIV>Thanks</DIV>
<DIV><BR></DIV>
<DIV> </DIV>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<DIV bgcolor="#FFFFFF" text="#000000">I was wondering if any of you guys know what could contribute to this difference.<SPAN><FONT color=#888888><BR><BR>Pranith</FONT></SPAN>
<DIV>
<DIV><BR>
<DIV>On 08/07/2014 01:33 AM, Anand Avati wrote:<BR></DIV>
<BLOCKQUOTE class=cite type="cite">
<DIV dir=ltr>Seems like heavy FINODELK contention. As a diagnostic step, can you try disabling eager-locking and check the write performance again (gluster volume set $name cluster.eager-lock off)?</DIV>
<DIV class=gmail_extra><BR><BR>
<DIV class=gmail_quote>On Tue, Aug 5, 2014 at 11:44 AM, David F. Robinson <SPAN dir=ltr><<A href="mailto:david.robinson@corvidtec.com" moz-do-not-send="true">david.robinson@corvidtec.com</A>></SPAN> wrote:<BR>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<DIV>
<DIV>Forgot to attach profile info in previous email. Attached... </DIV><SPAN><FONT color=#888888>
<DIV> </DIV>
<DIV>David</DIV></FONT></SPAN>
<DIV>
<DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>------ Original Message ------</DIV>
<DIV>From: "David F. Robinson" <<A href="mailto:david.robinson@corvidtec.com" moz-do-not-send="true">david.robinson@corvidtec.com</A>></DIV>
<DIV>To: <A href="mailto:gluster-devel@gluster.org" moz-do-not-send="true">gluster-devel@gluster.org</A></DIV>
<DIV>Sent: 8/5/2014 2:41:34 PM</DIV>
<DIV>Subject: Fw: Re: Corvid gluster testing</DIV>
<DIV> </DIV>
<DIV>
<BLOCKQUOTE class=cite cite=http://eme3fc1a31-df17-4978-a99c-f4b348e690f3@dfrobins-vaio type="cite">
<DIV>I have been testing some of the fixes that Pranith incorporated into the 3.5.2-beta to see how they performed for moderate levels of i/o. All of the stability issues that I had seen in previous versions seem to have been fixed in 3.5.2; however, there still seem to be some significant performance issues. Pranith suggested that I send this to the gluster-devel email list, so here goes: </DIV>
<DIV> </DIV>
<DIV>I am running an MPI job that saves a restart file to the gluster file system. When I use the following in my fstab to mount the gluster volume, the i/o time for the 2.5GB file is roughly 45-seconds.</DIV>
<DIV> </DIV>
<DIV><EM><FONT size=+0> gfsib01a.corvidtec.com:/homegfs /homegfs glusterfs transport=tcp,_netdev 0 0<BR></FONT></EM></DIV>
<DIV>When I switch this to use the NFS protocol (see below), the i/o time is 2.5-seconds.</DIV>
<DIV> </DIV>
<DIV><EM><FONT size=+0> gfsib01a.corvidtec.com:/homegfs /homegfs nfs vers=3,intr,bg,rsize=32768,wsize=32768 0 0</FONT></EM></DIV>
<DIV> </DIV>
<DIV>The read-times for gluster are 10-20% faster than NFS, but the write times are almost 20x slower. </DIV>
<DIV> </DIV>
<DIV>I am running SL 6.4 and glusterfs-3.5.2-0.1.beta1.el6.x86_64... </DIV>
<DIV> </DIV>
<DIV><EM><FONT size=+0>[root@gfs01a glusterfs]# gluster volume info homegfs<BR>Volume Name: homegfs<BR>Type: Distributed-Replicate<BR>Volume ID: 1e32672a-f1b7-4b58-ba94-58c085e59071<BR>Status: Started<BR>Number of Bricks: 2 x 2 = 4<BR>Transport-type: tcp<BR>Bricks:<BR>Brick1: gfsib01a.corvidtec.com:/data/brick01a/homegfs<BR>Brick2: gfsib01b.corvidtec.com:/data/brick01b/homegfs<BR>Brick3: gfsib01a.corvidtec.com:/data/brick02a/homegfs<BR>Brick4: gfsib01b.corvidtec.com:/data/brick02b/homegfs</FONT></EM><BR></DIV>
<DIV> </DIV>
<DIV>David</DIV>
<DIV> </DIV>
<DIV>------ Forwarded Message ------</DIV>
<DIV>From: "Pranith Kumar Karampuri" <<A href="mailto:pkarampu@redhat.com" moz-do-not-send="true">pkarampu@redhat.com</A>></DIV>
<DIV>To: "David Robinson" <<A href="mailto:david.robinson@corvidtec.com" moz-do-not-send="true">david.robinson@corvidtec.com</A>></DIV>
<DIV>Cc: "Young Thomas" <<A href="mailto:tom.young@corvidtec.com" moz-do-not-send="true">tom.young@corvidtec.com</A>></DIV>
<DIV>Sent: 8/5/2014 2:25:38 AM</DIV>
<DIV>Subject: Re: Corvid gluster testing</DIV>
<DIV> </DIV>
<DIV>
<DIV><A href="mailto:gluster-devel@gluster.org" moz-do-not-send="true">gluster-devel@gluster.org</A> is the email-id for the mailing list. We should probably start with the initial run numbers and the comparison for glusterfs mount and nfs mounts. May be something like </DIV>
<DIV> </DIV>
<DIV>glusterfs mount: 90 minutes </DIV>
<DIV>nfs mount: 25 minutes </DIV>
<DIV> </DIV>
<DIV>And profile outputs, volume config, number of mounts, hardware configuration should be a good start. </DIV>
<DIV> </DIV>
<DIV>Pranith </DIV>
<DIV> </DIV>
<DIV>On 08/05/2014 09:28 AM, David Robinson wrote: </DIV>
<BLOCKQUOTE class=cite type="cite">
<DIV>Thanks pranith </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>=============================== </DIV>
<DIV>David F. Robinson, Ph.D. </DIV>
<DIV>President - Corvid Technologies </DIV>
<DIV><A href="tel:704.799.6944%20x101" moz-do-not-send="true" value="+17047996944">704.799.6944 x101</A> [office] </DIV>
<DIV><A href="tel:704.252.1310" moz-do-not-send="true" value="+17042521310">704.252.1310</A> [cell] </DIV>
<DIV><A href="tel:704.799.7974" moz-do-not-send="true" value="+17047997974">704.799.7974</A> [fax] </DIV>
<DIV><A href="mailto:David.Robinson@corvidtec.com" moz-do-not-send="true">David.Robinson@corvidtec.com</A> </DIV>
<DIV><A href="http://www.corvidtechnologies.com/" moz-do-not-send="true">http://www.corvidtechnologies.com/</A> </DIV>
<DIV> </DIV>
<BLOCKQUOTE class=cite type="cite">
<DIV>On Aug 4, 2014, at 11:22 PM, Pranith Kumar Karampuri <<A href="mailto:pkarampu@redhat.com" moz-do-not-send="true">pkarampu@redhat.com</A>> wrote: </DIV>
<DIV> </DIV>
<DIV> </DIV>
<BLOCKQUOTE class=cite type="cite">
<DIV>On 08/05/2014 08:33 AM, Pranith Kumar Karampuri wrote: </DIV>
<DIV> </DIV>
<DIV>On 08/05/2014 08:29 AM, David F. Robinson wrote: </DIV>
<BLOCKQUOTE class=cite type="cite">
<BLOCKQUOTE class=cite type="cite">
<BLOCKQUOTE class=cite type="cite">
<DIV>On 08/05/2014 12:51 AM, David F. Robinson wrote: </DIV>
<DIV>No. I don't want to use nfs. It eliminates most of the benefits of why I want to use gluster. Failover redundancy of the pair, load balancing, etc. </DIV></BLOCKQUOTE>
<DIV>What is the meaning of 'Failover redundancy of the pair, load balancing ' Could you elaborate more? smb/nfs/glusterfs are just access protocols that gluster supports functionality is almost same </DIV></BLOCKQUOTE>
<DIV>Here is my understanding. Please correct me where I am wrong. </DIV>
<DIV> </DIV>
<DIV>With gluster, if I am doing a write and one of the replicated pairs goes down, there is no interruption to the I/o. The failover is handled by gluster and the fuse client. This isn't done if I use an nfs mount unless the component of the pair that goes down isn't the one I used for the mount. </DIV>
<DIV> </DIV>
<DIV>With nfs, I will have to mount one of the bricks. So, if I have gfs01a, gfs01b, gfs02a, gfs02b, gfs03a, gfs03b, etc and my fstab mounts gfs01a, it is my understanding that all of my I/o will go through gfs01a which then gets distributed to all of the other bricks. Gfs01a throughput becomes a bottleneck. Where if I do a gluster mount using fuse, the load balancing is handled at the client side , not the server side. If I have 1000-nodes accessing 20-gluster bricks, I need the load balancing aspect. I cannot have all traffic going through the network interface on a single brick. </DIV>
<DIV> </DIV>
<DIV>If I am wrong with the above assumptions, I guess my question is why would one ever use the gluster mount instead of nfs and/or samba? </DIV>
<DIV> </DIV>
<DIV>Tom: feel free to chime in if I have missed anything. </DIV></BLOCKQUOTE>
<DIV>I see your point now. Yes the gluster server where you did the mount is kind of a bottle neck. </DIV></BLOCKQUOTE>
<DIV>Now that we established the problem is in the clients/protocols, you should send out a detailed mail on gluster-devel and see if anyone can help with you on performance xlators that can improve it a bit more. My area of expertise is more on replication. I am sub-maintainer for replication,locks components. I also know connection management/io-threads related issues which lead to hangs as I worked on them before. Performance xlators are black box to me. </DIV>
<DIV> </DIV>
<DIV>Performance xlators are enabled only on fuse gluster stack. On nfs server mounts we disable all the performance xlators except write-behind as nfs client does lots of things for improving performance. I suggest you guys follow up more on gluster-devel. </DIV>
<DIV> </DIV>
<DIV>Appreciate all the help you did for improving the product :-). Thanks a ton! </DIV>
<DIV>Pranith </DIV>
<BLOCKQUOTE class=cite type="cite">
<DIV>Pranith </DIV>
<BLOCKQUOTE class=cite type="cite">
<DIV>David (Sent from mobile) </DIV>
<DIV> </DIV>
<DIV>=============================== </DIV>
<DIV>David F. Robinson, Ph.D. </DIV>
<DIV>President - Corvid Technologies </DIV>
<DIV><A href="tel:704.799.6944%20x101" moz-do-not-send="true" value="+17047996944">704.799.6944 x101</A> [office] </DIV>
<DIV><A href="tel:704.252.1310" moz-do-not-send="true" value="+17042521310">704.252.1310</A> [cell] </DIV>
<DIV><A href="tel:704.799.7974" moz-do-not-send="true" value="+17047997974">704.799.7974</A> [fax] </DIV>
<DIV><A href="mailto:David.Robinson@corvidtec.com" moz-do-not-send="true">David.Robinson@corvidtec.com</A> </DIV>
<DIV><A href="http://www.corvidtechnologies.com/" moz-do-not-send="true">http://www.corvidtechnologies.com/</A> </DIV></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE>
<DIV> </DIV></DIV></BLOCKQUOTE></DIV></DIV></DIV></DIV><BR>_______________________________________________<BR>Gluster-devel mailing list<BR><A href="mailto:Gluster-devel@gluster.org" moz-do-not-send="true">Gluster-devel@gluster.org</A><BR><A href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" moz-do-not-send="true">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</A><BR><BR></BLOCKQUOTE></DIV><BR></DIV><BR>
<FIELDSET></FIELDSET> <BR><PRE>_______________________________________________
Gluster-devel mailing list
<A href="mailto:Gluster-devel@gluster.org" moz-do-not-send="true">Gluster-devel@gluster.org</A>
<A href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" moz-do-not-send="true">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</A>
</PRE></BLOCKQUOTE><BR></DIV></DIV></DIV></BLOCKQUOTE></DIV><BR></DIV></DIV></BLOCKQUOTE><BR></DIV></DIV></DIV></BLOCKQUOTE></DIV><BR></DIV></BLOCKQUOTE><BR></BLOCKQUOTE></DIV></BODY></HTML>