<div dir="ltr"><div class="gmail_extra">On Mon, May 27, 2013 at 5:20 PM, Charles Cooke <span dir="ltr"><<a href="mailto:charles@coupa.com" target="_blank">charles@coupa.com</a>></span> wrote:<br><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>We are currently attempting to use GlusterFS but are encountering significant performance issues in our environment. The symptoms we are seeing are that read/write performance degrades and the glusterfsd eventually takes up to 100% of the CPU.</div>
<div><br></div><div>We currently run in a simple 1x2 replicated configuration. The I/O primarily consists of write operations, with a smaller amount of reads.</div><div><br></div><div>What we see consistently on profiling is that the LOOKUP calls are consistently taking the majority of the time. I am not sure what this represents - but the ordinality seems to be out of whack - eg. when several dozen read/write operations are taking place, there will be several thousand LOOKUP calls.<br>
</div><div><br></div><div>Following the discussion in the IRC Archives - there was some indication that our symptoms (large number of LOOKUP calls) were the result of self-heal checks and/or DHT misses as per:</div>
<div><a href="http://irclog.perlgeek.de/gluster/2012-11-28" target="_blank">http://irclog.perlgeek.de/gluster/2012-11-28</a><br></div><div><span style="white-space:pre-wrap">--> </span>JoeJulian<span style="white-space:pre-wrap">        </span>Well, that's either self-heal checks and/or dht misses.<br>
</div><div><br></div><div>So we've disabled the self-heal daemon and turned lookup-unhashed off. But this hasn't had an impact on the performance either way. The volume information is shown below along with volume profiling information.</div>
<div><div> </div><div>Volume Name: gluster</div><div>Type: Replicate</div><div>Volume ID: 08cb1e95-1f56-449c-b427-8f1921371d5a</div><div>Status: Started</div><div>Number of Bricks: 1 x 2 = 2</div><div>Transport-type: tcp</div>
<div>Bricks:</div><div>Brick1: stg100mol1-int.coupahost.com:/mnt/gluster/export</div><div>Brick2: stg100mol2-int.coupahost.com:/mnt/gluster/export</div><div>Options Reconfigured:</div><div>performance.cache-size: 2000000000</div>
<div>performance.cache-refresh-timeout: 0</div><div>geo-replication.indexing: off</div><div>diagnostics.latency-measurement: on</div><div>diagnostics.count-fop-hits: on</div><div>cluster.self-heal-daemon: off</div><div>cluster.lookup-unhashed: off</div>
<div><br></div><div><br></div><div><div>Brick: stg100mol1-int.coupahost.com:/mnt/gluster/export</div><div>-------------------------------------------------------</div><div>Cumulative Stats:</div><div> Block Size: 1b+ 2b+ 4b+</div>
<div> No. of Reads: 0 0 0</div><div>No. of Writes: 98 8 69</div><div><br></div><div> Block Size: 8b+ 16b+ 32b+</div>
<div> No. of Reads: 0 0 0</div><div>No. of Writes: 125 43 7</div><div><br></div><div> Block Size: 64b+ 128b+ 256b+</div>
<div> No. of Reads: 3 0 18</div><div>No. of Writes: 154 620 262</div><div><br></div><div> Block Size: 512b+ 1024b+ 2048b+</div>
<div> No. of Reads: 27 68 14</div><div>No. of Writes: 373 86 60</div><div><br></div><div> Block Size: 4096b+ 8192b+ 16384b+</div>
<div> No. of Reads: 0 6 14</div><div>No. of Writes: 11957 2398 6</div><div><br></div><div> Block Size: 32768b+ 65536b+ 131072b+</div>
<div> No. of Reads: 17 65 149</div><div>No. of Writes: 7 0 0</div><div><br></div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div>
<div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 712 FORGET</div><div> 0.00 0.00 us 0.00 us 0.00 us 5789 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us 5896 RELEASEDIR</div><div> 0.00 86.33 us 52.00 us 137.00 us 3 GETXATTR</div><div> 0.00 1045.00 us 1045.00 us 1045.00 us 1 RMDIR</div>
<div> 0.00 156.98 us 32.00 us 1272.00 us 41 FTRUNCATE</div><div> 0.00 572.24 us 212.00 us 3865.00 us 49 MKDIR</div><div> 0.01 135.47 us 36.00 us 6126.00 us 471 TRUNCATE</div>
<div> 0.01 160.58 us 14.00 us 5641.00 us 470 STAT</div><div> 0.03 517.82 us 38.00 us 163774.00 us 471 SETATTR</div><div> 0.07 1445.84 us 25.00 us 97172.00 us 396 READ</div>
<div> 0.07 401.11 us 13.00 us 33647.00 us 1449 READDIR</div><div> 0.10 3617.81 us 111.00 us 237368.00 us 221 STATFS</div><div> 0.10 8252.17 us 14.00 us 406281.00 us 99 LK</div>
<div> 0.10 1161.94 us 162.00 us 263151.00 us 709 CREATE</div><div> 0.12 2204.37 us 79.00 us 175090.00 us 457 RENAME</div><div> 0.13 1084.25 us 84.00 us 487294.00 us 942 XATTROP</div>
<div> 0.24 336.51 us 23.00 us 753796.00 us 5896 OPENDIR</div><div> 0.25 343.05 us 25.00 us 267306.00 us 5951 OPEN</div><div> 0.26 763.13 us 12.00 us 498524.00 us 2729 INODELK</div>
<div> 0.26 518.81 us 11.00 us 574717.00 us 4100 ENTRYLK</div><div> 0.27 5757.93 us 65.00 us 266003.00 us 377 UNLINK</div><div> 0.28 141.80 us 25.00 us 33487.00 us 16273 WRITE</div>
<div> 0.38 303.52 us 12.00 us 905949.00 us 10244 FSTAT</div><div> 0.88 14109.27 us 1733.00 us 574868.00 us 505 FSYNC</div><div> 2.17 1793.98 us 7.00 us 1341172.00 us 9841 FLUSH</div>
<div> 3.17 788.24 us 66.00 us 925523.00 us 32627 FXATTROP</div><div> 13.92 7132.57 us 55.00 us 735763.00 us 15853 READDIRP</div><div> 18.04 2726.17 us 7.00 us 1337047.00 us 53762 FINODELK</div>
<div> 59.14 630.92 us 29.00 us 1358867.00 us 761500 LOOKUP</div></div><div><br></div><div><div> Duration: 2281 seconds </div><div> Data Read: 27416867 bytes </div><div>Data Written: 115562427 bytes </div>
<div><br></div><div>Interval 47 Stats:</div><div> Block Size: 4096b+ 8192b+</div><div> No. of Reads: 0 0</div><div>No. of Writes: 104 21</div>
<div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 7 RELEASE</div>
<div> 0.00 0.00 us 0.00 us 0.00 us 21 RELEASEDIR</div><div> 0.00 18.86 us 13.00 us 36.00 us 7 FLUSH</div><div> 0.00 38.00 us 15.00 us 67.00 us 4 ENTRYLK</div>
<div> 0.00 230.00 us 230.00 us 230.00 us 1 MKDIR</div><div> 0.00 251.00 us 251.00 us 251.00 us 1 CREATE</div><div> 0.01 116.50 us 33.00 us 455.00 us 6 OPEN</div>
<div> 0.01 133.83 us 23.00 us 260.00 us 6 STAT</div><div> 0.02 827.50 us 139.00 us 1516.00 us 2 STATFS</div><div> 0.03 103.95 us 31.00 us 284.00 us 21 OPENDIR</div>
<div> 0.07 339.07 us 66.00 us 907.00 us 14 READDIRP</div><div> 0.08 89.11 us 20.00 us 750.00 us 64 FSTAT</div><div> 0.23 123.34 us 48.00 us 901.00 us 125 WRITE</div>
<div> 0.41 105.09 us 16.00 us 2021.00 us 263 FINODELK</div><div> 0.86 234.20 us 73.00 us 8309.00 us 249 FXATTROP</div><div> 98.27 794.98 us 32.00 us 372846.00 us 8420 LOOKUP</div>
<div><br></div><div> Duration: 11 seconds</div><div> Data Read: 0 bytes</div><div>Data Written: 993600 bytes</div><div><br></div></div><div><div>Brick: stg100mol2-int.coupahost.com:/mnt/gluster/export</div><div>-------------------------------------------------------</div>
<div>Cumulative Stats:</div><div> Block Size: 1b+ 2b+ 4b+</div><div> No. of Reads: 0 0 0</div><div>No. of Writes: 98 8 69</div>
<div><br></div><div> Block Size: 8b+ 16b+ 32b+</div><div> No. of Reads: 0 0 0</div><div>No. of Writes: 125 43 7</div>
<div><br></div><div> Block Size: 64b+ 128b+ 256b+</div><div> No. of Reads: 0 3 13</div><div>No. of Writes: 154 620 262</div>
<div><br></div><div> Block Size: 512b+ 1024b+ 2048b+</div><div> No. of Reads: 32 58 15</div><div>No. of Writes: 373 86 60</div>
<div><br></div><div> Block Size: 4096b+ 8192b+ 16384b+</div><div> No. of Reads: 0 6 16</div><div>No. of Writes: 11957 2398 6</div>
<div><br></div><div> Block Size: 32768b+ 65536b+ 131072b+</div><div> No. of Reads: 19 46 163</div><div>No. of Writes: 7 0 0</div>
<div><br></div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div><div> 0.00 0.00 us 0.00 us 0.00 us 712 FORGET</div>
<div> 0.00 0.00 us 0.00 us 0.00 us 5789 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 5896 RELEASEDIR</div><div> 0.00 131.00 us 131.00 us 131.00 us 1 RMDIR</div>
<div> 0.00 87.00 us 61.00 us 113.00 us 2 GETXATTR</div><div> 0.00 197.12 us 32.00 us 2304.00 us 41 FTRUNCATE</div><div> 0.01 602.80 us 214.00 us 11089.00 us 49 MKDIR</div>
<div> 0.01 105.21 us 38.00 us 4226.00 us 471 TRUNCATE</div><div> 0.01 514.50 us 17.00 us 44276.00 us 98 LK</div><div> 0.03 369.86 us 14.00 us 35614.00 us 322 STAT</div>
<div> 0.07 618.82 us 38.00 us 230853.00 us 471 SETATTR</div><div> 0.07 427.43 us 172.00 us 21070.00 us 709 CREATE</div><div> 0.13 589.38 us 107.00 us 47276.00 us 942 XATTROP</div>
<div> 0.15 3001.27 us 111.00 us 600028.00 us 221 STATFS</div><div> 0.18 284.42 us 11.00 us 230739.00 us 2725 INODELK</div><div> 0.24 2732.04 us 23.00 us 85370.00 us 385 READ</div>
<div> 0.27 2572.42 us 75.00 us 256620.00 us 457 RENAME</div><div> 0.33 239.45 us 24.00 us 625901.00 us 5951 OPEN</div><div> 0.37 169.74 us 15.00 us 433133.00 us 9356 FSTAT</div>
<div> 0.39 1164.77 us 14.00 us 468495.00 us 1449 READDIR</div><div> 0.44 5045.44 us 121.00 us 89320.00 us 377 UNLINK</div><div> 0.46 339.88 us 21.00 us 632652.00 us 5896 OPENDIR</div>
<div> 0.55 145.86 us 25.00 us 123050.00 us 16273 WRITE</div><div> 0.76 803.77 us 11.00 us 857743.00 us 4100 ENTRYLK</div><div> 1.69 14534.79 us 1493.00 us 464961.00 us 505 FSYNC</div>
<div> 4.71 2078.22 us 6.00 us 818633.00 us 9841 FLUSH</div><div> 5.05 2310.54 us 56.00 us 615643.00 us 9497 READDIRP</div><div> 5.95 791.32 us 63.00 us 608936.00 us 32628 FXATTROP</div>
<div> 6.97 563.13 us 6.00 us 1103183.00 us 53723 FINODELK</div><div> 71.18 405.92 us 8.00 us 1216554.00 us 761502 LOOKUP</div><div><br></div><div> Duration: 2250 seconds</div>
<div> Data Read: 27138474 bytes</div><div>Data Written: 115562427 bytes</div></div><div><br></div><div><div>Interval 47 Stats:</div><div> Block Size: 4096b+ 8192b+</div><div> No. of Reads: 0 0</div>
<div>No. of Writes: 104 21</div><div> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div> --------- ----------- ----------- ----------- ------------ ----</div>
<div> 0.00 0.00 us 0.00 us 0.00 us 7 RELEASE</div><div> 0.00 0.00 us 0.00 us 0.00 us 21 RELEASEDIR</div><div> 0.00 26.00 us 26.00 us 26.00 us 1 STAT</div>
<div> 0.00 195.00 us 195.00 us 195.00 us 1 CREATE</div><div> 0.00 53.25 us 27.00 us 86.00 us 4 ENTRYLK</div><div> 0.00 249.00 us 249.00 us 249.00 us 1 MKDIR</div>
<div> 0.00 159.00 us 125.00 us 193.00 us 2 STATFS</div><div> 0.01 52.00 us 18.00 us 143.00 us 7 FLUSH</div><div> 0.01 65.67 us 33.00 us 136.00 us 6 OPEN</div>
<div> 0.03 98.14 us 25.00 us 405.00 us 21 OPENDIR</div><div> 0.04 50.35 us 18.00 us 202.00 us 57 FSTAT</div><div> 0.17 381.32 us 77.00 us 926.00 us 28 READDIRP</div>
<div> 0.26 134.29 us 50.00 us 1403.00 us 125 WRITE</div><div> 0.93 240.55 us 71.00 us 10168.00 us 250 FXATTROP</div><div> 0.98 240.87 us 15.00 us 34353.00 us 264 FINODELK</div>
<div> 97.56 749.24 us 28.00 us 296964.00 us 8417 LOOKUP</div><div><br></div><div> Duration: 11 seconds</div><div> Data Read: 0 bytes</div><div>Data Written: 993600 bytes</div></div><span class="HOEnZb"><font color="#888888"><div>
<br>
</div><div><br></div><div><br></div><div><br></div><div dir="ltr"><div></div></div>
</font></span></div></div>
</blockquote></div><br>Hello,</div><div class="gmail_extra"><br></div><div class="gmail_extra">As a followup for anyone experiencing similar issues - switching to an NFS mount from a Gluster Native resolved the issue. </div>
<div class="gmail_extra"><br></div><div class="gmail_extra" style>The giveaway was the large number of context switches happening in the kernel during some heavy operations (vmstat or dstat) - neither the I/O, network interface, or CPU was the bottleneck - but the context switching was causing significant impact on the performance of our Java application. </div>
<div class="gmail_extra" style><br></div><div class="gmail_extra" style>The application has a large volume of writes on many small files, which is not ideal for the FUSE client. Switching to NFS brought the times down an order of magnitude.</div>
<div class="gmail_extra"><div dir="ltr"><div></div><div><br></div><div></div></div>
</div></div>