<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Help please-<div><br></div><div><br></div><div>I am running 3.3.1 on Centos using a 10GB network. &nbsp;I get reasonable write speeds, although I think they could be faster. &nbsp;But my read speeds are REALLY slow.</div><div><br></div><div>Executive summary:</div><div><br></div><div>On gluster client-</div><div>Writes average about 700-800MB/s</div><div>Reads average about 70-80MB/s</div><div><br></div><div>On server-</div><div>Writes average about 1-1.5GB/s</div><div>Reads average about 2-3GB/s</div><div><br></div><div>Any thoughts?</div><div><br></div><div><br></div><div><br></div><div>Here are some additional details:</div><div><br></div><div>Nothing interesting in any of the log files, everything is very quite.</div><div>All servers had no other load, and all clients are performing the same way.</div><div><br></div><div><br></div><div><div>Volume Name: shared</div><div>Type: Distribute</div><div>Volume ID: de11cc19-0085-41c3-881e-995cca244620</div><div>Status: Started</div><div>Number of Bricks: 26</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: fs-disk2:/storage/disk2a</div><div>Brick2: fs-disk2:/storage/disk2b</div><div>Brick3: fs-disk2:/storage/disk2d</div><div>Brick4: fs-disk2:/storage/disk2e</div><div>Brick5: fs-disk2:/storage/disk2f</div><div>Brick6: fs-disk2:/storage/disk2g</div><div>Brick7: fs-disk2:/storage/disk2h</div><div>Brick8: fs-disk2:/storage/disk2i</div><div>Brick9: fs-disk2:/storage/disk2j</div><div>Brick10: fs-disk2:/storage/disk2k</div><div>Brick11: fs-disk2:/storage/disk2l</div><div>Brick12: fs-disk2:/storage/disk2m</div><div>Brick13: fs-disk2:/storage/disk2n</div><div>Brick14: fs-disk2:/storage/disk2o</div><div>Brick15: fs-disk2:/storage/disk2p</div><div>Brick16: fs-disk2:/storage/disk2q</div><div>Brick17: fs-disk2:/storage/disk2r</div><div>Brick18: fs-disk2:/storage/disk2s</div><div>Brick19: fs-disk2:/storage/disk2t</div><div>Brick20: fs-disk2:/storage/disk2u</div><div>Brick21: fs-disk2:/storage/disk2v</div><div>Brick22: fs-disk2:/storage/disk2w</div><div>Brick23: fs-disk2:/storage/disk2x</div><div>Brick24: fs-disk3:/storage/disk3a</div><div>Brick25: fs-disk3:/storage/disk3b</div><div>Brick26: fs-disk3:/storage/disk3c</div><div>Options Reconfigured:</div><div>performance.write-behind: on</div><div>performance.read-ahead: on</div><div>performance.io-cache: on</div><div>performance.stat-prefetch: on</div><div>performance.quick-read: on</div><div>cluster.min-free-disk: 500GB</div><div>nfs.disable: off</div></div><div><br></div><div><br></div><div>sysctl.conf settings for 10GBe</div><div><div># increase TCP max buffer size settable using setsockopt()</div><div>net.core.rmem_max = 67108864&nbsp;</div><div>net.core.wmem_max = 67108864&nbsp;</div><div># increase Linux autotuning TCP buffer limit</div><div>net.ipv4.tcp_rmem = 4096 87380 67108864</div><div>net.ipv4.tcp_wmem = 4096 65536 67108864</div><div># increase the length of the processor input queue</div><div>net.core.netdev_max_backlog = 250000</div><div># recommended default congestion control is htcp&nbsp;</div><div>net.ipv4.tcp_congestion_control=htcp</div><div># recommended for hosts with jumbo frames enabled</div><div>net.ipv4.tcp_mtu_probing=1</div></div><div><br></div><div><br></div><div><div><br></div></div><div><br></div><br><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Thomas W.<br>Sr. &nbsp;Systems Administrator COLA/IGES<br><a href="mailto:twake@cola.iges.org">twake@cola.iges.org</a><br></div></div></span></span></div>Affiliate&nbsp;<span class="Apple-style-span" style="font-size: 12px; ">Computer Scientist GMU</span></span>
</div>
<br></body></html>