<div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">Hello!</span><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">I have GlusterFS installation with parameters:</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">- 4 servers, connected by 1Gbit/s network (760-800 Mbit/s by iperf)</div><div style="font-family:arial,sans-serif;font-size:13px">
- Distributed-replicated volume with 4 bricks and 2x4 redundancy formula.</div><div style="font-family:arial,sans-serif;font-size:13px">- Replicated volume with 2 bricks and 2x2 formula.</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">I found some trouble: if I try to copy huge amount of files (94000 files, 3Gb size), this process takes terribly long time (from 20 to 40 minutes). I perform some tests and results is:</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Directly to storage (single 2TB HDD): 158MB/s</div><div style="font-family:arial,sans-serif;font-size:13px">
Directly to storage (RAID1 of 2 HDDs): 190MB/s</div><div style="font-family:arial,sans-serif;font-size:13px">To Replicated gluster volume: 89MB/s</div><div style="font-family:arial,sans-serif;font-size:13px">To Distributed-replicated gluster volume: 49MB/s<br clear="all">
<div><br></div><div>Test command is: sync && echo 3 > /proc/sys/vm/drop_caches && dd if=/dev/zero of=gluster.test.bin bs=1G count=1</div><div><br></div><div>Switching direct-io on and off doesn't have effect.</div>
<div>Playing with glusterfs options too.</div><div><br></div><div>What I can do with performance?</div><div><br></div><div>My volumes:</div><div><br></div><div><div>Volume Name: nginx</div><div>Type: Replicate</div><div>Volume ID: e3306431-e01d-41f8-8b2d-86a61837b0b2</div>
<div>Status: Started</div><div>Number of Bricks: 1 x 2 = 2</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: control1:/storage/nginx</div><div>Brick2: control2:/storage/nginx</div><div> </div><div>Volume Name: instances</div>
<div>Type: Distributed-Replicate</div><div>Volume ID: d32363fc-4b53-433c-87b7-ad51acfa4125</div><div>Status: Started</div><div>Number of Bricks: 2 x 2 = 4</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: control1:/storage/instances</div>
<div>Brick2: control2:/storage/instances</div><div>Brick3: compute1:/storage/instances</div><div>Brick4: compute2:/storage/instances</div><div>Options Reconfigured:</div><div>cluster.self-heal-window-size: 1</div><div>cluster.data-self-heal-algorithm: diff</div>
<div>performance.stat-prefetch: 1</div><div>features.quota-timeout: 3600</div><div>performance.write-behind-window-size: 512MB</div><div>performance.cache-size: 1GB</div><div>performance.io-thread-count: 64</div><div>performance.flush-behind: on</div>
<div>performance.cache-min-file-size: 0</div><div>performance.write-behind: on</div><div><br></div><div>Mounted with default options by Gluster-FUSE.</div></div></div><br clear="all"><div><br></div>-- <br>With best regards,<br>
differentlocal (<a href="http://www.differentlocal.ru">www.differentlocal.ru</a> | <a href="mailto:differentlocal@gmail.com">differentlocal@gmail.com</a>),<br>System administrator.
</div>