Hello,<br> We redid our setup with ext4 instead of xfs - and it seems to work fine. It sounds like there may be a bug in the integration of xfs + glusterfs. Thanks.<br><br><br><div class="gmail_quote">On Fri, Jul 6, 2012 at 11:03 AM, Khawaja Shams <span dir="ltr"><<a href="mailto:kshams@usc.edu" target="_blank">kshams@usc.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">Hello,<br> Thanks for the response. Does our create volume look correct? <br>1) We are using XFS<br>2) We are using: <pre>
dd if=/dev/zero of=/mnt/home/testfile bs=16k count=16384</pre></div>to
write a 256MB file filled with 0s. I don't think that is a sparse file.
We also tried other methods of writing the file to the system, and each file is
consistently 40 times larger than we expect it to be. Should we try a
different fs? <br><div class="im">
<br>3) We haven't tried the cluster.stripe-coalesce option. We will
investigate it now. Google didn't turn up any useful info on how to use
that at a first glance, but we will look a little further. <br><br><br>Thank you. <br><br></div><div class="gmail_quote"><div class="im">On Fri, Jul 6, 2012 at 9:13 AM, Jeff Darcy <span dir="ltr"><<a href="mailto:jdarcy@redhat.com" target="_blank">jdarcy@redhat.com</a>></span> wrote:<br>
</div><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On 07/06/2012 11:29 AM, Khawaja Shams wrote:<br>
> Hello,<br>
> We are trying to create a 40 volume gluster across 4 machines. For the<br>
> purposes of our benchmarks, we are trying to set it up without any replication.<br>
> However, files are taking up 40 times the storage space than we think. Putting<br>
> out a 256MB file with dd puts a 9.2GB file on the file system. The gluster FAQs<br>
> claim that ls doesn't track the file space properly and du does:<br>
> <a href="http://gluster.org/community/documentation//index.php/GlusterFS_Technical_FAQ#Stripe_behavior_not_working_as_expected" target="_blank">http://gluster.org/community/documentation//index.php/GlusterFS_Technical_FAQ#Stripe_behavior_not_working_as_expected</a><br>
><br>
> However, in our case, ls shows the file to be 256MB, while du shows it to be<br>
> 9.2GB. After looking at the individual drives, we also noticed that each drive<br>
> has 256MB of data and it seems to be getting replicated. Here is how we create<br>
> the volumes:<br>
><br>
> gluster create volume gluster-test stripe 40 transport tcp Gluster1:/data_f<br>
> Gluster1:/data_g Gluster1:/data_h Gluster1:/data_i Gluster1:/data_j<br>
> Gluster1:/data_k Gluster1:/data_l Gluster1:/data_m Gluster1:/data_n<br>
> Gluster1:/data_o Gluster2:/data_f Gluster2:/data_g Gluster2:/data_h<br>
> Gluster2:/data_i Gluster2:/data_j Gluster2:/data_k Gluster2:/data_l<br>
> Gluster2:/data_m Gluster2:/data_n Gluster2:/data_o Gluster3:/data_f<br>
> Gluster3:/data_g Gluster3:/data_h Gluster3:/data_i Gluster3:/data_j<br>
> Gluster3:/data_k Gluster3:/data_l Gluster3:/data_m Gluster3:/data_n<br>
> Gluster3:/data_o Gluster4:/data_f Gluster4:/data_g Gluster4:/data_h<br>
> Gluster4:/data_i Gluster4:/data_j Gluster4:/data_k Gluster4:/data_l<br>
> Gluster4:/data_m Gluster4:/data_n Gluster4:/data_o<br>
><br>
> gluster --version<br>
</div>> *glusterfs 3.3.0 built on Jul 2 2012 23:26:48*<br>
> Repository revision: git://<a href="http://git.gluster.com/glusterfs.git" target="_blank">git.gluster.com/glusterfs.git</a><br>
> <<a href="http://git.gluster.com/glusterfs.git" target="_blank">http://git.gluster.com/glusterfs.git</a>><br>
<div>> Copyright (c) 2006-2011 Gluster Inc. <<a href="http://www.gluster.com" target="_blank">http://www.gluster.com</a>><br>
> GlusterFS comes with ABSOLUTELY NO WARRANTY.<br>
> You may redistribute copies of GlusterFS under the terms of the GNU General<br>
> Public License.<br>
><br>
> We do not have this issue if all the bricks are on the same machine. What are<br>
> we doing wrong? Are there specific logs we should be looking at?<br>
<br>
</div>Three questions come to mind.<br>
<br>
(1) What underlying local filesystem (e.g. XFS, ext4) are you using for the bricks?<br>
<br>
(2) What exactly is the "dd" command you're using? Block size is particularly<br>
important, also whether you're trying to write a sparse file.<br>
<br>
(3) Have you tried turning on the "cluster.stripe-coalesce" volume option, and<br>
did it make a difference?<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
</blockquote></div></div></div><br>
</blockquote></div><br>