<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=GB2312" http-equiv=Content-Type>
<STYLE>
BLOCKQUOTE {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px; MARGIN-LEFT: 2em
}
OL {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
UL {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
P {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
BODY {
LINE-HEIGHT: 1.5; FONT-FAMILY: ËÎÌå; COLOR: #000080; FONT-SIZE: 10.5pt
}
</STYLE>
<META name=GENERATOR content="MSHTML 9.00.8112.16457"></HEAD>
<BODY style="MARGIN: 10px">
<DIV>Thanks all the responses.</DIV>
<DIV>It seems that I need to furtherly describe our question. If our
VM is mounted on gluster via NFS(V3), writing big data on VM
reach the full bandwidth. However, if VM is mounted by
gluster client, it has half bandwidth when writing data on VM (mounted
on gluster). </DIV>
<DIV>Considering we don't want VM users see our gluster file system, mount
gluster file on VM is not allowed.</DIV>
<DIV>By the way, Where can I get the 3.4a version of Gluster? </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>Regards,</DIV>
<HR style="WIDTH: 210px; HEIGHT: 1px" align=left color=#b5c4df SIZE=1>
<DIV><SPAN>zhxue</SPAN></DIV>
<DIV> </DIV>
<DIV
style="BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0cm; PADDING-LEFT: 0cm; PADDING-RIGHT: 0cm; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<DIV
style="PADDING-BOTTOM: 8px; PADDING-LEFT: 8px; PADDING-RIGHT: 8px; BACKGROUND: #efefef; COLOR: #000000; FONT-SIZE: 12px; PADDING-TOP: 8px">
<DIV><B>From:</B> <A
href="mailto:gluster-users-request@gluster.org">gluster-users-request</A></DIV>
<DIV><B>Date:</B> 2013-01-14 20:00</DIV>
<DIV><B>To:</B> <A
href="mailto:gluster-users@gluster.org">gluster-users</A></DIV>
<DIV><B>Subject:</B> Gluster-users Digest, Vol 57, Issue
31</DIV></DIV></DIV>
<DIV>
<DIV>Send Gluster-users mailing list submissions to</DIV>
<DIV>gluster-users@gluster.org</DIV>
<DIV> </DIV>
<DIV>To subscribe or unsubscribe via the World Wide Web, visit</DIV>
<DIV>http://supercolony.gluster.org/mailman/listinfo/gluster-users</DIV>
<DIV>or, via email, send a message with subject or body 'help' to</DIV>
<DIV>gluster-users-request@gluster.org</DIV>
<DIV> </DIV>
<DIV>You can reach the person managing the list at</DIV>
<DIV>gluster-users-owner@gluster.org</DIV>
<DIV> </DIV>
<DIV>When replying, please edit your Subject line so it is more specific</DIV>
<DIV>than "Re: Contents of Gluster-users digest..."</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>Today's Topics:</DIV>
<DIV> </DIV>
<DIV> 1. IO performance cut down when VM on Gluster (glusterzhxue)</DIV>
<DIV> 2. Re: IO performance cut down when VM on Gluster (Joe Julian)</DIV>
<DIV> 3. Re: IO performance cut down when VM on Gluster</DIV>
<DIV> (Stephan von Krawczynski)</DIV>
<DIV> 4. Re: IO performance cut down when VM on Gluster (Bharata B Rao)</DIV>
<DIV> 5. dm-glusterfs (was Re: IO performance cut down when VM on</DIV>
<DIV> Gluster) (Jeff Darcy)</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>----------------------------------------------------------------------</DIV>
<DIV> </DIV>
<DIV>Message: 1</DIV>
<DIV>Date: Sun, 13 Jan 2013 20:14:36 +0800</DIV>
<DIV>From: glusterzhxue <glusterzhxue@163.com></DIV>
<DIV>To: gluster-users <gluster-users@gluster.org></DIV>
<DIV>Subject: [Gluster-users] IO performance cut down when VM on Gluster</DIV>
<DIV>Message-ID: <2013011320143501335810@163.com></DIV>
<DIV>Content-Type: text/plain; charset="gb2312"</DIV>
<DIV> </DIV>
<DIV>Hi all,</DIV>
<DIV>We placed Virtual Machine Imame(based on kvm) on gluster file system, but IO performance of the VM is only half of the bandwidth.</DIV>
<DIV>If we mount it on a physical machine using the same volume as the above VM, physical host reaches full bandwidth. We performed it many times, but each had the same result.</DIV>
<DIV>Anybody could help us?</DIV>
<DIV> </DIV>
<DIV>Thanks</DIV>
<DIV> </DIV>
<DIV>zhxue</DIV>
<DIV>-------------- next part --------------</DIV>
<DIV>An HTML attachment was scrubbed...</DIV>
<DIV>URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130113/78161071/attachment-0001.html></DIV>
<DIV> </DIV>
<DIV>------------------------------</DIV>
<DIV> </DIV>
<DIV>Message: 2</DIV>
<DIV>Date: Sun, 13 Jan 2013 07:11:14 -0800</DIV>
<DIV>From: Joe Julian <joe@julianfamily.org></DIV>
<DIV>To: gluster-users@gluster.org</DIV>
<DIV>Subject: Re: [Gluster-users] IO performance cut down when VM on</DIV>
<DIV>Gluster</DIV>
<DIV>Message-ID: <50F2CE92.6060703@julianfamily.org></DIV>
<DIV>Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"</DIV>
<DIV> </DIV>
<DIV>On 01/13/2013 04:14 AM, glusterzhxue wrote:</DIV>
<DIV>> Hi all,</DIV>
<DIV>> We placed Virtual Machine Imame(based on kvm) on gluster file system, </DIV>
<DIV>> but IO performance of the VM is only half of the bandwidth.</DIV>
<DIV>> If we mount it on a physical machine using the same volume as the </DIV>
<DIV>> above VM, physical host reaches full bandwidth. We performed it many </DIV>
<DIV>> times, but each had the same result.</DIV>
<DIV>What you're seeing is the difference between bandwidth and latency. When </DIV>
<DIV>you're writing a big file to a VM filesystem, you're not performing the </DIV>
<DIV>same operations as writing a file to a GlusterFS mount thus you're able </DIV>
<DIV>to measure bandwidth. The filesystem within the VM is doing things like </DIV>
<DIV>journaling, inode operations, etc. that you don't have to do when </DIV>
<DIV>writing to the client requiring a lot more I/O operations per second, </DIV>
<DIV>thus amplifying the latency present in both your network and the context </DIV>
<DIV>switching through FUSE.</DIV>
<DIV> </DIV>
<DIV>You have two options:</DIV>
<DIV>1. Mount the GlusterFS volume from within the VM and host the data </DIV>
<DIV>you're operating on there. This avoids all the additional overhead of </DIV>
<DIV>managing a filesystem on top of FUSE.</DIV>
<DIV>2. Try the 3.4 qa release and native GlusterFS support in the latest </DIV>
<DIV>qemu-kvm.</DIV>
<DIV> </DIV>
<DIV>-------------- next part --------------</DIV>
<DIV>An HTML attachment was scrubbed...</DIV>
<DIV>URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130113/7cc6641e/attachment-0001.html></DIV>
<DIV> </DIV>
<DIV>------------------------------</DIV>
<DIV> </DIV>
<DIV>Message: 3</DIV>
<DIV>Date: Sun, 13 Jan 2013 23:55:01 +0100</DIV>
<DIV>From: Stephan von Krawczynski <skraw@ithnet.com></DIV>
<DIV>To: Joe Julian <joe@julianfamily.org></DIV>
<DIV>Cc: gluster-users@gluster.org</DIV>
<DIV>Subject: Re: [Gluster-users] IO performance cut down when VM on</DIV>
<DIV>Gluster</DIV>
<DIV>Message-ID: <20130113235501.c0a2eb24.skraw@ithnet.com></DIV>
<DIV>Content-Type: text/plain; charset=US-ASCII</DIV>
<DIV> </DIV>
<DIV>On Sun, 13 Jan 2013 07:11:14 -0800</DIV>
<DIV>Joe Julian <joe@julianfamily.org> wrote:</DIV>
<DIV> </DIV>
<DIV>> On 01/13/2013 04:14 AM, glusterzhxue wrote:</DIV>
<DIV>> > Hi all,</DIV>
<DIV>> > We placed Virtual Machine Imame(based on kvm) on gluster file system, </DIV>
<DIV>> > but IO performance of the VM is only half of the bandwidth.</DIV>
<DIV>> > If we mount it on a physical machine using the same volume as the </DIV>
<DIV>> > above VM, physical host reaches full bandwidth. We performed it many </DIV>
<DIV>> > times, but each had the same result.</DIV>
<DIV>> What you're seeing is the difference between bandwidth and latency. When </DIV>
<DIV>> you're writing a big file to a VM filesystem, you're not performing the </DIV>
<DIV>> same operations as writing a file to a GlusterFS mount thus you're able </DIV>
<DIV>> to measure bandwidth. The filesystem within the VM is doing things like </DIV>
<DIV>> journaling, inode operations, etc. that you don't have to do when </DIV>
<DIV>> writing to the client requiring a lot more I/O operations per second, </DIV>
<DIV>> thus amplifying the latency present in both your network and the context </DIV>
<DIV>> switching through FUSE.</DIV>
<DIV>> </DIV>
<DIV>> You have two options:</DIV>
<DIV>> 1. Mount the GlusterFS volume from within the VM and host the data </DIV>
<DIV>> you're operating on there. This avoids all the additional overhead of </DIV>
<DIV>> managing a filesystem on top of FUSE.</DIV>
<DIV>> 2. Try the 3.4 qa release and native GlusterFS support in the latest </DIV>
<DIV>> qemu-kvm.</DIV>
<DIV> </DIV>
<DIV>Thank you for telling the people openly that FUSE is a performance problem</DIV>
<DIV>which could be solved by a kernel-based glusterfs.</DIV>
<DIV> </DIV>
<DIV>Do you want to make drivers for every application like qemu? How many burnt</DIV>
<DIV>manpower will it take until the real solution is accepted?</DIV>
<DIV>It is no solution to mess around _inside_ the VM for most people, you simply</DIV>
<DIV>don't want _customers_ on your VM with a glusterfs mount. You want them to see</DIV>
<DIV>a local fs only.</DIV>
<DIV> </DIV>
<DIV>-- </DIV>
<DIV>Regards,</DIV>
<DIV>Stephan</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>------------------------------</DIV>
<DIV> </DIV>
<DIV>Message: 4</DIV>
<DIV>Date: Mon, 14 Jan 2013 09:55:53 +0530</DIV>
<DIV>From: Bharata B Rao <bharata.rao@gmail.com></DIV>
<DIV>To: Stephan von Krawczynski <skraw@ithnet.com></DIV>
<DIV>Cc: gluster-users@gluster.org</DIV>
<DIV>Subject: Re: [Gluster-users] IO performance cut down when VM on</DIV>
<DIV>Gluster</DIV>
<DIV>Message-ID:</DIV>
<DIV><CAGZKiBr--fYF-Awq0cYXJx1wPB52Odgm_PArE3Dvrt733mfwZw@mail.gmail.com></DIV>
<DIV>Content-Type: text/plain; charset=ISO-8859-1</DIV>
<DIV> </DIV>
<DIV>On Mon, Jan 14, 2013 at 4:25 AM, Stephan von Krawczynski</DIV>
<DIV><skraw@ithnet.com> wrote:</DIV>
<DIV>></DIV>
<DIV>> Thank you for telling the people openly that FUSE is a performance problem</DIV>
<DIV>> which could be solved by a kernel-based glusterfs.</DIV>
<DIV>></DIV>
<DIV>> Do you want to make drivers for every application like qemu? How many burnt</DIV>
<DIV>> manpower will it take until the real solution is accepted?</DIV>
<DIV>> It is no solution to mess around _inside_ the VM for most people, you simply</DIV>
<DIV>> don't want _customers_ on your VM with a glusterfs mount. You want them to see</DIV>
<DIV>> a local fs only.</DIV>
<DIV> </DIV>
<DIV>Just wondering if there is a value in doing dm-glusterfs on the lines</DIV>
<DIV>similar to dm-nfs</DIV>
<DIV>(https://blogs.oracle.com/OTNGarage/entry/simplify_your_storage_management_with).</DIV>
<DIV> </DIV>
<DIV>I understand GlusterFS due to its stackable translator nature and</DIV>
<DIV>having to deal with multiple translators at the client end might not</DIV>
<DIV>easily fit to this model, but may be something to think about ?</DIV>
<DIV> </DIV>
<DIV>Regards,</DIV>
<DIV>Bharata.</DIV>
<DIV>-- </DIV>
<DIV>http://raobharata.wordpress.com/</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>------------------------------</DIV>
<DIV> </DIV>
<DIV>Message: 5</DIV>
<DIV>Date: Mon, 14 Jan 2013 06:53:58 -0500</DIV>
<DIV>From: Jeff Darcy <jdarcy@redhat.com></DIV>
<DIV>To: gluster-users@gluster.org</DIV>
<DIV>Subject: [Gluster-users] dm-glusterfs (was Re: IO performance cut down</DIV>
<DIV>when VM on Gluster)</DIV>
<DIV>Message-ID: <50F3F1D6.10405@redhat.com></DIV>
<DIV>Content-Type: text/plain; charset=ISO-8859-1</DIV>
<DIV> </DIV>
<DIV>On 1/13/13 11:25 PM, Bharata B Rao wrote:</DIV>
<DIV>> Just wondering if there is a value in doing dm-glusterfs on the lines</DIV>
<DIV>> similar to dm-nfs</DIV>
<DIV>> (https://blogs.oracle.com/OTNGarage/entry/simplify_your_storage_management_with).</DIV>
<DIV>> </DIV>
<DIV>> I understand GlusterFS due to its stackable translator nature and</DIV>
<DIV>> having to deal with multiple translators at the client end might not</DIV>
<DIV>> easily fit to this model, but may be something to think about ?</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>It's an interesting idea. You're also right that there are some issues with</DIV>
<DIV>the stackable translator model and so on. Porting all of that code into the</DIV>
<DIV>kernel would require an almost suicidal suspension of all other development</DIV>
<DIV>activity while competitors continue to catch up on manageability or add other</DIV>
<DIV>features, so that's not every appealing. Keeping it all out in user space with</DIV>
<DIV>a minimal kernel-interception layer would give us something better than FUSE (I</DIV>
<DIV>did something like this in a previous life BTW), but probably not enough better</DIV>
<DIV>to be compelling. A hybrid "fast path, slow path" approach might work. Keep</DIV>
<DIV>all of the code for common-case reads and writes in the kernel, punt everything</DIV>
<DIV>else back up to user space with hooks to disable the fast path when necessary</DIV>
<DIV>(e.g. during a config change). OTOH, how would this be better than e.g. an</DIV>
<DIV>iSCSI target, which is deployable today with essentially the same functionality</DIV>
<DIV>and even greater generality (e.g. to non-Linux platforms)?</DIV>
<DIV> </DIV>
<DIV>It's good to think about these things. We could implement ten other</DIV>
<DIV>alternative access mechanisms (Apache/nginx modules anyone?) and still burn</DIV>
<DIV>fewer resources than we would with "just put it all in the kernel" inanity. I</DIV>
<DIV>tried one of our much-touted alternatives recently and, despite having a kernel</DIV>
<DIV>client, they achieved less than 1/3 of our performance on this kind of</DIV>
<DIV>workload. If we want to eliminate sources of overhead we need to address more</DIV>
<DIV>than just that one.</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>------------------------------</DIV>
<DIV> </DIV>
<DIV>_______________________________________________</DIV>
<DIV>Gluster-users mailing list</DIV>
<DIV>Gluster-users@gluster.org</DIV>
<DIV>http://supercolony.gluster.org/mailman/listinfo/gluster-users</DIV>
<DIV> </DIV>
<DIV>End of Gluster-users Digest, Vol 57, Issue 31</DIV>
<DIV>*********************************************</DIV></DIV></BODY></HTML>