<div dir="ltr">Yes, I'd recommend sticking with RAID in addition to GlusterFS. The cluster I'm mid-build on (it's a live migration) is 18x RAID-5 bricks on 9 servers. Each RAID-5 brick is 8 2T drives, so about 13T usable. It's better to deal with a RAID when a disk fails than to have to pull and replace the brick, and I believe Red Hat's official recommendation is still to minimize the number of bricks per server (which makes me a rebel for having two, I suppose). 9 (slow-ish, SATA RAID) servers easily saturate 1Gbit on a busy day.<div>
<br></div><div> The following is opinion only, so make up your own mind:</div><div><br></div><div>If I had a big pile of RAID-5 or RAID-6 bricks, I would not want to spend extra money for replica-3. Instead, I would go replica-2 and use the leftover money to build in additional redundancy on the hardware (e.g. redundant power, redundant 10gigE). If money were not an object, of course there's no harm in going replica-3 or more. But every build I've ever done has a budget that seems slightly small for the desired outcome.</div>
<div> </div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Dec 30, 2013 at 5:54 AM, bernhard glomm <span dir="ltr"><<a href="mailto:bernhard.glomm@ecologic.eu" target="_blank">bernhard.glomm@ecologic.eu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">some years ago I had a similar tasks.<div>I did:</div><div>- We had disk arrays with 24 slots, with optional 4 JBODS (each 24 slots) stacked on top, dual LWL controller 4GB (costs ;-) </div>
<div>- creating raids (6) with not more than 7 disks each</div><div>- as far as I remember I had one hot spare per each 4 raids</div><div>- connecting as many of this raid bricks together with striped glusterfs as needed</div>
<div>- as for replication, I was planing for an offside duplicate of this architecture and</div><div>because losing data was REALLY not an option, writing it all off at a second offside location onto LTFS tapes.</div><div>
As the original version for the LTFS library edition was far to expensive for us </div><div>I found an alternative solution that does the same thing</div><div>but fort a much reasonable prize. LTFS is still a big thing in digital Archiving.</div>
<div>Give me a note if you like more details on that.</div><div><br></div><div>- This way I could fsck all (not to big) raids in parallel (sped things up)</div><div>- proper robustness against disk failure</div><div>- space that could grow infinite in size (add more and bigger disks) and keep up with access speed (ad more server) at a pretty foreseeable prize</div>
<div>- LTFS in the vault provided just the finishing having data accessible even if two out three sides are down, </div><div>reasonable prize, (for instance no heat problem at the tape location)</div><div>Nowadays I would go for the same approach except zfs raidz3 bricks (at least do a thorough test on it) </div>
<div>instead of (small) hardware raid bricks. </div><div>As for simplicity and robustness I wouldn't like to end up with several hundred glusterfs bricks, each on one individual disk,</div><div>but rather leaving disk failure prevention either to hardware raid or zfs and using gluster to connect this bricks into the</div>
<div>fs size I need( - and for mirroring the whole thing to a second side if needed)</div><div>hth</div><div>Bernhard</div><div><br></div><div><br></div><div><br></div><div><div><div style><table border="0" cellpadding="0" cellspacing="5">
<tbody>
<tr>
<td align="center" style="color:#182983;text-align:center">
<img alt="*Ecologic Institute*" height="120" width="129">
</td>
<td align="left" style="text-align:left;padding-right:10px;white-space:nowrap">
<font face="sans-serif" color="black">
<small>
<b>Bernhard Glomm</b><br>
IT Administration<br><br>
</small>
<small><small>
<table border="0" cellpadding="0" cellspacing="0">
<tbody><tr>
<td style="white-space:nowrap"><small><small>
Phone:
</small></small></td>
<td style="padding-left:5px;white-space:nowrap"><small><small>
<a href="tel:%2B49%20%2830%29%2086880%20134" value="+493086880134" target="_blank">+49 (30) 86880 134</a>
</small></small></td>
</tr>
<tr>
<td style="white-space:nowrap"><small><small>
Fax:
</small></small></td>
<td style="padding-left:5px;white-space:nowrap"><small><small>
<a href="tel:%2B49%20%2830%29%2086880%20100" value="+493086880100" target="_blank">+49 (30) 86880 100</a>
</small></small></td>
</tr>
<tr>
<td style="white-space:nowrap"><small><small>
Skype:
</small></small></td>
<td style="padding-left:5px;white-space:nowrap"><small><small>
bernhard.glomm.ecologic
</small></small></td>
</tr>
</tbody></table>
</small></small>
</font>
</td>
</tr>
<tr>
<td colspan="2" valign="center" align="center" style="vertical-align:middle;text-align:center;background-color:#6f7072;color:#ffffff;white-space:nowrap">
<a href="http://ecologic.eu" target="_blank"><img title="Visit our website" alt="Website:" height="36" width="36"></a>
<a href="http://www.youtube.com/v/hZtiK04A9Yo" target="_blank"><img title="Watch our video" alt="| Video:" height="36" width="36"></a>
<a href="http://ecologic.eu/newsletter/subscribe" target="_blank"><img title="Subscribe to our newsletter" alt="| Newsletter:" height="36" width="36"></a>
<a href="http://www.facebook.com/Ecologic.Institute" target="_blank"><img title="Visit us on Facebook" alt="| Facebook:" height="36" width="36"></a>
<a href="http://www.linkedin.com/company/ecologic-institute-berlin-germany" target="_blank"><img title="Visit us on Linkedin" alt="| Linkedin:" height="36" width="36"></a>
<a href="http://twitter.com/EcologicBerlin" target="_blank"><img title="Visit us on Twitter" alt="| Twitter:" height="36" width="36"></a>
<a href="http://www.youtube.com/user/EcologicInstitute" target="_blank"><img title="Visit us on YouTube" alt="| YouTube:" height="36" width="36"></a>
<a href="http://plus.google.com/113756356645020994482" target="_blank"><img title="Visit us on Google+" alt="| Google+:" height="36" width="36"></a>
</td>
</tr>
<tr>
<td colspan="2" valign="center" align="left" style="padding-right:5px;padding-left:5px;vertical-align:middle;text-align:left;white-space:nowrap">
<small><small>
Ecologic Institut gemeinnützige GmbH | Pfalzburger Str. 43/44 | 10717 Berlin | Germany<br>
GF: R. Andreas Kraemer | AG: Charlottenburg HRB 57947 | USt/VAT-IdNr.: DE811963464<br>
Ecologic™ is a Trade Mark (TM) of Ecologic Institut gemeinnützige GmbH
</small></small>
</td>
</tr>
<tr>
<td colspan="2" valign="center" align="center" style="vertical-align:middle;color:#ffffff;text-align:center">
<hr style="min-height:2px;background-color:#6f7072;color:#6f7072;border-width:0">
</td>
</tr>
</tbody></table>
</div>
</div>
<br><div><div><div class="h5"><div>On Dec 25, 2013, at 8:47 PM, Fredrik Häll <<a href="mailto:hall.fredrik@gmail.com" target="_blank">hall.fredrik@gmail.com</a>> wrote:</div><br></div></div><blockquote type="cite">
<div><div class="h5"><div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">I am new to Gluster, but so far it seems very attractive for my needs. I am trying to assess its suitability for a cost-efficient storage problem I am tackling. Hopefully someone can help me find how to best solve my problem. </span><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">Capacity: </div><div style="font-family:arial,sans-serif;font-size:13px">Start with around 0.5PB usable</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">Redundancy: </div><div style="font-family:arial,sans-serif;font-size:13px">2 replicas with non-RAID is not sufficient. Either 3 replicas with non-raid or some combination of 2 replicas and RAID?</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">File types: </div><div style="font-family:arial,sans-serif;font-size:13px">Large files, around 400-1500MB each. </div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Usage pattern: </div><div style="font-family:arial,sans-serif;font-size:13px">Archive (not sure if this matches nearline or not..) with files being added at around 200-300GB/day (3-400 files/day). Very few reads, order of 10 file accesses per day. Concurrent reads highly unlikely. </div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">The main two factors for me are cost and redundancy. Losing data is not an option, being an archive solution. Cost/usable TB is the other key factor, as we see growth estimates of 100-500TB/year.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Looking just at $/TB, a RAID-based approach to me sounds more efficient. But RAID rebuild times with large arrays of large capacity drives sound really scary. Not sure if something smart can be done since we will still have a replica left during the rebuild?</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">So, any suggestions on what would be possible and cost-efficient solutions? </div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">- Any experience on dense servers, what is advisable? 24/36/50/60 slots?</div><div style="font-family:arial,sans-serif;font-size:13px">- SAS expanders/storage pods?</div>
<div style="font-family:arial,sans-serif;font-size:13px">- RAID vs non-RAID?</div><div style="font-family:arial,sans-serif;font-size:13px">- Number of replicas etc? </div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">Best, </div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Fredrik</div></div>
</div></div>
_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div><br></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Justin Dossey<br>
CTO, PodOmatic<div><br></div>
</div>