<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>... keep it simple, make it robust ...</div><div>use raid1 (or raidz if you can) for the bricks</div><div>hth</div><div><br>On 19.02.2014, at 20:32, Targino Silveira <<a href="mailto:targinosilveira@gmail.com">targinosilveira@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">Sure, <div><br></div><div>I will use XFS, as I sayd before it's for old data, so we don't need a great performance, we only need to store data.</div><div><br></div><div>regards,</div></div><div class="gmail_extra">
<br clear="all"><div><div dir="ltr">Targino Silveira<br>+55-85-8626-7297<br><a href="http://www.twitter.com/targinosilveira" target="_blank">www.twitter.com/targinosilveira</a><br></div></div>
<br><br><div class="gmail_quote">2014-02-19 16:11 GMT-03:00 BGM <span dir="ltr"><<a href="mailto:bernhard.glomm@ecologic.eu" target="_blank">bernhard.glomm@ecologic.eu</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto"><div>well, note:</div><div>- you don't need zfs on the hardeware machines, xfs or ext3 or ext4 would do it too</div><div>- for production you wouldn't use a glusterfs on top of a glusterfs but rather giving the vm access to a real blockdevice, like a whole harddisk or at least a partition of it although migration of the vm wouldn't be possible than...</div>
<div>therefor: a VM as a glusterserver might not be the best idea.</div><div>- remember to peer probe the glusterserver partner from both sides! as mentioned below</div><div><br></div><div>for a first setup you should be fine with that.</div>
<div><br></div><div>regards</div><div><div class="h5"><div><br>On 19.02.2014, at 19:32, Targino Silveira <<a href="mailto:targinosilveira@gmail.com" target="_blank">targinosilveira@gmail.com</a>> wrote:<br><br></div>
<blockquote type="cite"><div><div dir="ltr">Thanks Bernhard I will do this.<div><br></div><div>Regards, </div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div dir="ltr">Targino Silveira<br><a href="tel:%2B55-85-8626-7297" value="+558586267297" target="_blank">+55-85-8626-7297</a><br>
<a href="http://www.twitter.com/targinosilveira" target="_blank">www.twitter.com/targinosilveira</a><br>
</div></div>
<br><br><div class="gmail_quote">2014-02-19 14:43 GMT-03:00 Bernhard Glomm <span dir="ltr"><<a href="mailto:bernhard.glomm@ecologic.eu" target="_blank">bernhard.glomm@ecologic.eu</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><font face="tahoma, arial, helvetica, sans-serif">I would strongly recommend to restart fresh with gluster 3.2.4 from <span></span></font><div><font face="tahoma, arial, helvetica, sans-serif"><a href="http://download.gluster.org/pub/gluster/glusterfs/3.4/" target="_blank">http://download.gluster.org/pub/gluster/glusterfs/3.4/</a></font></div>
<div><span style="font-family:tahoma,arial,helvetica,sans-serif">It works totally fine for me.</span></div><div><span style="font-family:tahoma,arial,helvetica,sans-serif">(</span><span style="font-family:tahoma,arial,helvetica,sans-serif">reinstall the vms as slim as possible if you can.)</span></div>
<div><br></div><div><font face="tahoma, arial, helvetica, sans-serif">As a quick howto consider this:</font></div><div><font face="tahoma, arial, helvetica, sans-serif"><br></font></div><div><div>- We have 2 Hardware machines (just desktop machines for dev-env)</div>
<div>- both running zol</div><div>- create a zpool and zfs filesystem</div><div>- create a gluster replica 2 volume between hostA and hostB</div><div>- installe 3 VM vmachine0{4,5,6}</div><div>- vmachine0{4,5} each have a 100GB diskimage file as /dev/vdb which also resides on the glustervolume</div>
<div>- create ext3 filesystem on vmachine0{4,5}:/dev/vdb1</div><div>- create gluster replica 2 between vmachine04 and vmachine05 as shown below</div><div><br></div><div><div>(!!!obviously nobody would do that in any serious environment,</div>
<div>just to show that even a setup like that _would_ be possible!!!)</div></div><div><br></div><div>- run some benchmarks on that volume and compare the results to other </div><div><br></div><div>So:</div><div><br></div>
<div>root@vmachine04[/0]:~ # mkdir -p /srv/vdb1/gf_brick</div><div>root@vmachine04[/0]:~ # mount /dev/vdb1 /srv/vdb1/</div><div>root@vmachine04[/0]:~ # gluster peer probe vmachine05</div><div>peer probe: success</div><div>
<br></div><div># now switch over to vmachine05 and do</div><div><br></div><div>root@vmachine05[/1]:~ # mkdir -p /srv/vdb1/gf_brick</div><div>root@vmachine05[/1]:~ # mount /dev/vdb1 /srv/vdb1/</div><div>root@vmachine05[/1]:~ # gluster peer probe vmachine04</div>
<div>peer probe: success</div><div>root@vmachine05[/1]:~ # gluster peer probe vmachine04</div><div>peer probe: success: host vmachine04 port 24007 already in peer list</div><div><br></div><div># the peer probe from BOTH sides ist often forgotten </div>
<div># switch back to vmachine04 and continue with</div><div><br></div><div>root@vmachine04[/0]:~ # gluster peer status</div><div>Number of Peers: 1</div><div><br></div><div>Hostname: vmachine05</div><div>Port: 24007</div>
<div>Uuid: 085a1489-dabf-40bb-90c1-fbfe66539953</div><div>State: Peer in Cluster (Connected)</div><div>root@vmachine04[/0]:~ # gluster volume info layer_cake_volume</div><div><br></div><div>Volume Name: layer_cake_volume</div>
<div>Type: Replicate</div><div>Volume ID: ef5299db-2896-4631-a2a8-d0082c1b25be</div><div>Status: Started</div><div>Number of Bricks: 1 x 2 = 2</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: vmachine04:/srv/vdb1/gf_brick</div>
<div>Brick2: vmachine05:/srv/vdb1/gf_brick</div><div>root@vmachine04[/0]:~ # gluster volume status layer_cake_volume</div><div>Status of volume: layer_cake_volume</div><div>Gluster process Port Online Pid</div>
<div>------------------------------------------------------------------------------</div><div>Brick vmachine04:/srv/vdb1/gf_brick 49152 Y 12778</div><div>Brick vmachine05:/srv/vdb1/gf_brick 49152 Y 16307</div>
<div>NFS Server on localhost 2049 Y 12790</div><div>Self-heal Daemon on localhost N/A Y 12791</div><div>NFS Server on vmachine05 2049 Y 16320</div>
<div>Self-heal Daemon on vmachine05 N/A Y 16319</div><div><br></div><div>There are no active volume tasks</div><div><br></div><div># set any option you might like</div><div><br></div>
<div>root@vmachine04[/1]:~ # gluster volume set layer_cake_volume network.remote-dio enable</div><div>volume set: success</div><div><br></div><div># go to vmachine06 and mount the volume</div><div>root@vmachine06[/1]:~ # mkdir /srv/layer_cake</div>
<div>root@vmachine06[/1]:~ # mount -t glusterfs -o backupvolfile-server=vmachine05 vmachine04:/layer_cake_volume /srv/layer_cake</div><div>root@vmachine06[/1]:~ # mount</div><div>vmachine04:/layer_cake_volume on /srv/layer_cake type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)</div>
<div>root@vmachine06[/1]:~ # df -h</div><div>Filesystem Size Used Avail Use% Mounted on</div><div>...</div><div>vmachine04:/layer_cake_volume 97G 188M 92G 1% /srv/layer_cake</div><div><br></div><div>
All fine and stable</div><div><br></div><div><br></div><div><br></div><div># now let's see how it tastes</div><div># note this is postmark on / NOT on the glustermounted layer_cake_volume!</div><div># that postmark results might be available tomorrow ;-)))</div>
<div>root@vmachine06[/1]:~ # postmark</div><div>PostMark v1.51 : 8/14/01</div><div>pm>set transactions 500000</div><div>pm>set number 200000</div><div>pm>set subdirectories 10000</div><div>pm>run</div><div>Creating subdirectories...Done</div>
<div>Creating files...Done</div><div>Performing transactions..........Done</div><div>Deleting files...Done</div><div>Deleting subdirectories...Done</div><div>Time:</div><div> 2314 seconds total</div><div> 2214 seconds of transactions (225 per second)</div>
</div><div><div>Files:</div><div> 450096 created (194 per second)</div><div> Creation alone: 200000 files (4166 per second)</div><div> Mixed with transactions: 250096 files (112 per second)</div>
<div> 249584 read (112 per second)</div><div> 250081 appended (112 per second)</div><div> 450096 deleted (194 per second)</div><div> Deletion alone: 200192 files (3849 per second)</div>
<div> Mixed with transactions: 249904 files (112 per second)</div><div><br></div><div>Data:</div><div> 1456.29 megabytes read (644.44 kilobytes per second)</div><div> 2715.89 megabytes written (1.17 megabytes per second)</div>
<div><br></div><div># reference</div><div># running postmark on the hardware machine directly on zfs</div><div>#</div><div># /test # postmark</div><div># PostMark v1.51 : 8/14/01</div><div># pm>set transactions 500000</div>
<div># pm>set number 200000</div><div># pm>set subdirectories 10000</div><div># pm>run</div><div># Creating subdirectories...Done</div><div># Creating files...Done</div>
<div># Performing transactions..........Done</div><div># Deleting files...Done</div><div># Deleting subdirectories...Done</div><div># Time:</div><div># 605 seconds total</div>
<div># 549 seconds of transactions (910 per second)</div><div>#</div><div># Files:</div><div># 450096 created (743 per second)</div><div># Creation alone: 200000 files (4255 per second)</div>
<div># Mixed with transactions: 250096 files (455 per second)</div><div># 249584 read (454 per second)</div><div># 250081 appended (455 per second)</div><div># 450096 deleted (743 per second)</div>
<div># Deletion alone: 200192 files (22243 per second)</div><div># Mixed with transactions: 249904 files (455 per second)</div><div>#</div><div># Data:</div><div># 1456.29 megabytes read (2.41 megabytes per second)</div>
<div># 2715.89 megabytes written (4.49 megabytes per second)</div><div><br></div><div>dbench -D /srv/layer_cake 5</div><div><br></div><div> Operation Count AvgLat MaxLat</div><div> ----------------------------------------</div>
<div> NTCreateX 195815 <a href="tel:5.159%20%C2%A0%20333.296" value="+555159333296" target="_blank">5.159 333.296</a></div><div> Close 143870 0.793 93.619</div><div> Rename 8310 10.922 123.096</div>
<div> Unlink 39525 <a href="tel:2.428%20%C2%A0%20203.753" value="+552428203753" target="_blank">2.428 203.753</a></div><div> Qpathinfo 177736 2.551 220.605</div><div> Qfileinfo 31030 2.057 175.565</div>
<div> Qfsinfo 32545 1.393 174.045</div><div> Sfileinfo 15967 2.691 129.028</div><div> Find 68664 <a href="tel:9.629%20%C2%A0%20185.739" value="+559629185739" target="_blank">9.629 185.739</a></div>
<div> WriteX 96860 0.841 108.863</div><div> ReadX 307834 0.511 213.602</div><div> LockX 642 1.511 10.578</div><div> UnlockX 642 1.541 10.137</div><div> Flush 13712 12.853 405.383</div>
<div><br></div><div>Throughput 10.1832 MB/sec 5 clients 5 procs max_latency=405.405 ms</div><div><br></div><div><br></div><div># reference</div><div>dbench -D /tmp 5</div></div><div><div># reference</div><div>dbench -D /tmp 5</div>
<div><br></div><div> Operation Count AvgLat MaxLat</div><div> ----------------------------------------</div><div> NTCreateX 3817455 0.119 499.847</div><div> Close 2804160 0.005 16.000</div>
<div> Rename 161655 0.322 459.790</div><div> Unlink 770906 0.556 762.314</div><div> Deltree 92 20.647 81.619</div><div> Mkdir 46 0.003 0.012</div><div> Qpathinfo 3460227 0.017 18.388</div>
<div> Qfileinfo 606258 0.003 11.652</div><div> Qfsinfo 634444 0.006 14.976</div><div> Sfileinfo 310990 0.155 604.585</div><div> Find 1337732 0.056 18.466</div><div> WriteX 1902611 0.245 503.604</div>
<div> ReadX 5984135 0.008 16.154</div><div> LockX 12430 0.008 9.111</div><div> UnlockX 12430 0.004 4.551</div><div> Flush 267557 4.505 902.093</div><div><br></div>
<div>Throughput 199.664 MB/sec 5 clients 5 procs max_latency=902.099 ms</div></div><div><br></div></div>
</blockquote></div><br></div>
</div></blockquote></div></div></div></blockquote></div><br></div>
</div></blockquote></body></html>