<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Dear Gluster Users,<br>
<br>
We are facing some severe performance issues with GlusterFS and we would very much appreciate any help on identifying the cause of this.<br>
<br>
Our setup is extremely simple: 2 nodes interconnected with 40Gb/s Infiniband and also 1Gb/s Ethernet, running Centos 6.2 and GlusterFS 3.2.6.<br>
Each node has 4 SATA drives put in a RAID0 array that gives ~750 MB/s random reads bandwidth. The tool that we used for measuring IO performance relies on O_DIRECT access, so we patched the fuse kernel:
<a href="http://marc.info/?l=linux-fsdevel&m=132950081331043&w=2" target="_blank">
http://marc.info/?l=linux-fsdevel&m=132950081331043&w=2</a>. <br>
<br>
We created the following volume and mounted it at <span style="font-style: italic;">
/mnt/gfs/</span>.<br>
<br>
<div style="margin-left: 40px;"><font size="2"><span style="font-family: Courier New;">Volume Name: GFS_RDMA_VOLUME</span><br style="font-family: Courier New;">
<span style="font-family: Courier New;">Type: Stripe</span><br style="font-family: Courier New;">
<span style="font-family: Courier New;">Status: Started</span><br style="font-family: Courier New;">
<span style="font-family: Courier New;">Number of Bricks: 2</span><br style="font-family: Courier New;">
<span style="font-family: Courier New;">Transport-type: rdma</span><br style="font-family: Courier New;">
<span style="font-family: Courier New;">Bricks:</span><br style="font-family: Courier New;">
<span style="font-family: Courier New;">Brick1: node01:/mnt/md0/gfs_storage</span><br style="font-family: Courier New;">
<span style="font-family: Courier New;">Brick2: node02:/mnt/md0/gfs_storage</span><br style="font-family: Courier New;">
<span style="font-family: Courier New;">Options Reconfigured:</span><br style="font-family: Courier New;">
<span style="font-family: Courier New;">cluster.stripe-block-size: *:2MB</span><br style="font-family: Courier New;">
<span style="font-family: Courier New;">performance.quick-read: on</span><br style="font-family: Courier New;">
<span style="font-family: Courier New;">performance.io-cache: on</span><br style="font-family: Courier New;">
<span style="font-family: Courier New;">performance.cache-size: 256MB</span><br style="font-family: Courier New;">
<span style="font-family: Courier New;">performance.cache-max-file-size: 128MB</span></font><br>
<br>
</div>
We expected to see an IO bandwidth of about 1500 MB/s (measured with the exact same tool and parameters), but unfortunately we only get ~100MB/s, which is very disappointing.<br>
<br>
Please find below the output of <span style="font-style: italic;">#cat /var/log/glusterfs/mnt-gfs-.log</span>. If you need any other information that I forgot to mentioned, please let me know.<br>
<br>
Thanks,<br>
Adrian<br>
<div style="margin-left: 40px;"><span style="font-family: Courier New;"><br>
</span></div>
<hr style="width: 100%; height: 2px;">
<span style="font-family: Courier New;"></span>
<div style="margin-left: 40px;"><span style="font-family: Courier New;"><font size="1"><span style="font-family: Courier New;">[2012-04-18 11:59:42.847818] I [glusterfsd.c:1493:main] 0-/opt/glusterfs/3.2.6/sbin/glusterfs: Started running /opt/glusterfs/3.2.6/sbin/glusterfs
version 3.2.6<br>
[2012-04-18 11:59:42.862610] W [write-behind.c:3023:init] 0-GFS_RDMA_VOLUME-write-behind: disabling write-behind for first 0 bytes<br>
[2012-04-18 11:59:43.318188] I [client.c:1935:notify] 0-GFS_RDMA_VOLUME-client-0: parent translators are ready, attempting connect on transport<br>
[2012-04-18 11:59:43.321287] I [client.c:1935:notify] 0-GFS_RDMA_VOLUME-client-1: parent translators are ready, attempting connect on transport<br>
Given volfile:<br>
+------------------------------------------------------------------------------+<br>
1: volume GFS_RDMA_VOLUME-client-0<br>
2: type protocol/client<br>
3: option remote-host node01<br>
4: option remote-subvolume /mnt/md0/gfs_storage<br>
5: option transport-type rdma<br>
6: end-volume<br>
7: <br>
8: volume GFS_RDMA_VOLUME-client-1<br>
9: type protocol/client<br>
10: option remote-host node02<br>
11: option remote-subvolume /mnt/md0/gfs_storage<br>
12: option transport-type rdma<br>
13: end-volume<br>
14: <br>
15: volume GFS_RDMA_VOLUME-stripe-0<br>
16: type cluster/stripe<br>
17: option block-size *:2MB<br>
18: subvolumes GFS_RDMA_VOLUME-client-0 GFS_RDMA_VOLUME-client-1<br>
19: end-volume<br>
20: <br>
21: volume GFS_RDMA_VOLUME-write-behind<br>
22: type performance/write-behind<br>
23: subvolumes GFS_RDMA_VOLUME-stripe-0<br>
24: end-volume<br>
25: <br>
26: volume GFS_RDMA_VOLUME-read-ahead<br>
27: type performance/read-ahead<br>
28: subvolumes GFS_RDMA_VOLUME-write-behind<br>
29: end-volume<br>
30: <br>
31: volume GFS_RDMA_VOLUME-io-cache<br>
32: type performance/io-cache<br>
33: option max-file-size 128MB<br>
34: option cache-size 256MB<br>
35: subvolumes GFS_RDMA_VOLUME-read-ahead<br>
36: end-volume<br>
37: <br>
38: volume GFS_RDMA_VOLUME-quick-read<br>
39: type performance/quick-read<br>
40: option cache-size 256MB<br>
41: subvolumes GFS_RDMA_VOLUME-io-cache<br>
42: end-volume<br>
43: <br>
44: volume GFS_RDMA_VOLUME-stat-prefetch<br>
45: type performance/stat-prefetch<br>
46: subvolumes GFS_RDMA_VOLUME-quick-read<br>
47: end-volume<br>
48: <br>
49: volume GFS_RDMA_VOLUME<br>
50: type debug/io-stats<br>
51: option latency-measurement off<br>
52: option count-fop-hits off<br>
53: subvolumes GFS_RDMA_VOLUME-stat-prefetch<br>
54: end-volume<br>
<br>
+------------------------------------------------------------------------------+<br>
[2012-04-18 11:59:43.326287] E [client-handshake.c:1171:client_query_portmap_cbk] 0-GFS_RDMA_VOLUME-client-1: failed to get the port number for remote subvolume<br>
[2012-04-18 11:59:43.764287] E [client-handshake.c:1171:client_query_portmap_cbk] 0-GFS_RDMA_VOLUME-client-0: failed to get the port number for remote subvolume<br>
[2012-04-18 11:59:46.868595] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-GFS_RDMA_VOLUME-client-0: changing port to 24009 (from 0)<br>
[2012-04-18 11:59:46.879292] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-GFS_RDMA_VOLUME-client-1: changing port to 24009 (from 0)<br>
[2012-04-18 11:59:50.872346] I [client-handshake.c:1090:select_server_supported_programs] 0-GFS_RDMA_VOLUME-client-0: Using Program GlusterFS 3.2.6, Num (1298437), Version (310)<br>
[2012-04-18 11:59:50.872760] I [client-handshake.c:913:client_setvolume_cbk] 0-GFS_RDMA_VOLUME-client-0: Connected to 192.168.0.101:24009, attached to remote volume '/mnt/md0/gfs_storage'.<br>
[2012-04-18 11:59:50.874975] I [client-handshake.c:1090:select_server_supported_programs] 0-GFS_RDMA_VOLUME-client-1: Using Program GlusterFS 3.2.6, Num (1298437), Version (310)<br>
[2012-04-18 11:59:50.875290] I [client-handshake.c:913:client_setvolume_cbk] 0-GFS_RDMA_VOLUME-client-1: Connected to 192.168.0.103:24009, attached to remote volume '/mnt/md0/gfs_storage'.<br>
[2012-04-18 11:59:50.878013] I [fuse-bridge.c:3339:fuse_graph_setup] 0-fuse: switched to graph 0<br>
[2012-04-18 11:59:50.878321] I [fuse-bridge.c:2927:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13</span></font><br>
</span></div>
<hr style="width: 100%; height: 2px;">
<span style="font-family: Courier New;"></span></div>
</body>
</html>