<br><br><div class="gmail_quote">On Tue, Mar 5, 2013 at 6:39 AM, Bharata B Rao <span dir="ltr"><<a href="mailto:bharata.rao@gmail.com" target="_blank">bharata.rao@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
Here is a highly experimental patch to support zero copy readv in<br>
GlusterFS. An attempt is made to copy the read data in the socket to<br>
client supplied buffers (iovecs) directly thereby eliminating one<br>
memory copy for each readv request. Currently I have support for zero<br>
copy readv only in glfs_preadv_async() which is what QEMU uses.<br>
<br>
The approach is basically a hack to quickly measure any performance<br>
gains that could be had with zero copy readv. I will work on a proper<br>
approach and come via gerrit after we decide to pursue this further<br>
for the kind of gains I am seeing. I have taken the path of least<br>
changes in the core rpc/socket code and path of least resistance in<br>
the xlator code to get the implementation working and I don't claim<br>
that this is the right or the optimal approach. And the volume is also<br>
pruned down so that I don't have to touch those xlators that aren't<br>
strictly needed.<br>
<br>
The volume configuration looks like this:<br>
<br>
[root@bharata ~]# gluster volume info<br>
<br>
Volume Name: test<br>
Type: Distribute<br>
Volume ID: 62461295-c265-4d15-916c-f283f28f6cbd<br>
Status: Started<br>
Number of Bricks: 1<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: bharata:/test<br>
Options Reconfigured:<br>
server.allow-insecure: on<br>
performance.read-ahead: off<br>
performance.write-behind: off<br>
performance.io-cache: off<br>
performance.quick-read: off<br>
<br>
I have used FIO from QEMU guest to compare the performance. The setup<br>
is as follows:<br>
Host: 2 core x86_64 sytem, 3.6.11-5.fc17.x86_64 kernel<br>
Guest: 4 CPU Fedora 17, 3.3.4-5.fc17.x86_64 kernel<br>
QEMU commit: 7ce4106c2125<br>
GlusterFS commit: 4e15a0b4189fe5<br>
QEMU cmdline: qemu-system-x86_64 --enable-kvm -nographic -m 2048 -smp<br>
4 -drive file=gluster://bharata/test/F17-test,if=virtio,cache=none<br>
FIO configuration:<br>
; Read 4 files with aio at different depths<br>
[global]<br>
ioengine=libaio<br>
direct=1<br>
rw=read<br>
bs=128k<br>
size=512m<br>
directory=/data1<br>
[file1]<br>
iodepth=4<br>
[file2]<br>
iodepth=32<br>
[file3]<br>
iodepth=8<br>
[file4]<br>
iodepth=16<br>
<br>
Avg of 10 FIO runs from guest with and without zero copy readv:<br>
<br>
Without zerocopy: 43817kB/s<br>
With zerocopy 46194kB/s ==> around 5% gain.<br></blockquote><div><br></div><div>Do you have comparison of the %cpu util with and without zero-copy? cpu util is probably be an important parameter of comparison.</div><div>
<br></div><div>Avati</div></div>