Coming back to the problem of quota sizes not converging, the converging may not complete if glusterd is restarted before converging process is complete because a thread is spawned by glusterd to do the converging. Hence, if glusterd is restarted converging will fail or to put it precisely it will not be completed. So running ls -lR on the mount point will solve the problem.<div>
<br><div class="gmail_quote">On Mon, Sep 5, 2011 at 4:40 PM, Mohammed Junaid <span dir="ltr"><<a href="mailto:junaid@gluster.com">junaid@gluster.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi Anthony,<div><br></div><div>This is quiet unusual, to debug this further can you get the output of "du -h" of mount-point and of directory soft (which is under mount point). This is will help a lot in debugging this issue. Also, I noticed that the volume info output provided is of a distributed-replicate setup which is different from the one provided previously (it was plain replicate). Also please provide the output of getfattr for the bricks and soft directory.</div>
<div><br></div><font color="#888888"><div>Junaid</div></font><div><div></div><div class="h5"><div><br><div class="gmail_quote">On Mon, Sep 5, 2011 at 2:58 PM, anthony garnier <span dir="ltr"><<a href="mailto:sokar6012@hotmail.com" target="_blank">sokar6012@hotmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div dir="ltr">
<br>Volume configuration: <br><br>yval1000:/ # gluster volume info<br><br>Volume Name: venus<br>Type: Distributed-Replicate<br>Status: Started<br>Number of Bricks: 2 x 2 = 4<div><br>Transport-type: tcp<br>Bricks:<br>
Brick1: ylal3020:/soft/venus<br>Brick2: ylal3030:/soft/venus<br></div>Brick3: yval1000:/soft/venus<br>Brick4: yval1010:/soft/venus<br>Options Reconfigured:<br>features.quota-timeout: 120<br>features.limit-usage: /test:100MB,/psa:200MB,/:7GB,/soft:5GB<br>
features.quota: on<br>geo-replication.indexing: on<br>network.ping-timeout: 10<br>performance.cache-size: 1GB<br>performance.cache-refresh-timeout: 60<br>nfs.port: 2049<br><br><br>Indeed values didn't converged automaticaly for big folder, for small folder like /test and /psa it was fine (I waited 1 day before checking the value). I was forced to run ls -lRa several times, but even with this workaround, values for big folder like /soft are not rightfully calculated : <br>
<div><br>gluster volume quota venus list<br> path limit_set size<br>----------------------------------------------------------------------------------<br></div>/test 100MB 4.5MB <= OK<br>
/psa 200MB 167.0MB <=OK<br>/ 7GB 5.2GB <=Strange<br>/soft 5GB 4.7GB <= Strange<br><br>=> the size of the entire filesystem is 1.4GB<br>
<br><br><br>Anthony<br><br><br><div><hr>From: <a href="mailto:junaid@gluster.com" target="_blank">junaid@gluster.com</a><br>Date: Mon, 5 Sep 2011 14:38:06 +0530<div><div></div><div><br>Subject: Re: [Gluster-users] quota calculation gluster 3.2.3<br>
To: <a href="mailto:sokar6012@hotmail.com" target="_blank">sokar6012@hotmail.com</a><br>CC: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br><br>Hi Anthony,<div><br></div><div>By volume configuration, I meant 'is the volume setup dht, afr, or dht-afr', I would like to know the output of </div>
<div><br></div><div> volume info volume-name</div><div><br></div>
<div>When data is already present before the activation of quota, it takes some time for quota values to converge because it has to crawl through the entire file-system to calculate the sizes. This is automatically done when you run</div>
<div><br></div><div> volume quota vol-name enable<br><br></div><div>The output of </div><div> volume quota vol-name list</div><div><br></div><div>will reflect the aggregated value at that point in time. If the values are not converging then its a issue. A simple work-around is to run ls -lR on the mount point. Please confirm if the values were not converging automatically, we will have a look into it.</div>
<div><br></div><div>Junaid</div><div><br><br><div>On Mon, Sep 5, 2011 at 12:51 PM, anthony garnier <span dir="ltr"><<a href="mailto:sokar6012@hotmail.com" target="_blank">sokar6012@hotmail.com</a>></span> wrote:<br>
<blockquote style="border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div><div dir="ltr">
Junaid,<br><br>What do you mean by " the volume configuration on other two servers" ? Configuration is the same on all the server..<br>I'm creating data with "dd" cmd, and yes, data was already present before activation of quota.<br>
<br><br><br><div><hr>From: <a href="mailto:junaid@gluster.com" target="_blank">junaid@gluster.com</a><br>Date: Fri, 2 Sep 2011 22:55:02 +0530<div><div></div><div><br>Subject: Re: [Gluster-users] quota calculation gluster 3.2.3<br>
To: <a href="mailto:sokar6012@hotmail.com" target="_blank">sokar6012@hotmail.com</a><br>CC: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br><br>Hi Anthony,<div><br></div><div>Thanks for providing the input. What is the volume configuration on other two servers (yval1000 and yval1010)? My question is same, what data where you creating on these two machines? Was the data already present before quota was enabled?<br>
<br><div>On Fri, Sep 2, 2011 at 1:49 PM, anthony garnier <span dir="ltr"><<a href="mailto:sokar6012@hotmail.com" target="_blank">sokar6012@hotmail.com</a>></span> wrote:<br><blockquote style="padding-left:1ex">
<div><div dir="ltr">
<br>
<div>
<div dir="ltr">
I've done a ls -lRa throught the mount point and now i get : <br>/soft/venus # gluster volume quota venus list<br> path limit_set size<br>----------------------------------------------------------------------------------<br>
/ 2GB 1.3GB<br>/test 100MB 8.0KB<br>/psa 200MB 151.0MB<br>/soft 1500MB 826.3MB<br>
<br>=>Seems to be consistent<br><br>and now command getfattr on yval1000 and yval1010 show : <br><br>yval1000 : <br>getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/ <br>trusted.glusterfs.quota.size=0x0000000028f9ca00<div>
<br><br>getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/soft<br></div>trusted.glusterfs.quota.size=0x000000001421d400<div><br><br>getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/psa<br>
</div>
trusted.glusterfs.quota.size=0x00000000053d6000<br><br> <br>yval1010 : <br>getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/ <br>trusted.glusterfs.quota.size=0x000000002904ea00<div><br><br>getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/soft<br>
</div>trusted.glusterfs.quota.size=0x000000002292ba00<div><br><br>getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/psa<br></div>trusted.glusterfs.quota.size=0x00000000053d6000<br><br>Regards,<br><br>
Anthony</div></div><br><br><div><div><hr>From: <a href="mailto:sokar6012@hotmail.com" target="_blank">sokar6012@hotmail.com</a><br>To: <a href="mailto:junaid@gluster.com" target="_blank">junaid@gluster.com</a><br>
CC: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>Subject: RE: [Gluster-users] quota calculation gluster 3.2.3<br></div>Date: Fri, 2 Sep 2011 07:57:19 +0000<div><div></div><div>
<br><br>
<div dir="ltr">
I forgot to check on the 2 other server.<br>So ylal3020 and ylal3030 show similar output send previously.<br><br>But yval1000 and yval1010 show with all command :<br> trusted.glusterfs.quota.size=0x0000000000000000<br><br>
Is it normal ?<br><br>Regards,<br><br>Anthony<br><div><hr>From: <a href="mailto:sokar6012@hotmail.com" target="_blank">sokar6012@hotmail.com</a><br>To: <a href="mailto:junaid@gluster.com" target="_blank">junaid@gluster.com</a><br>
CC: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>Subject: RE: [Gluster-users] quota calculation gluster 3.2.3<br>Date: Fri, 2 Sep 2011 07:51:52 +0000<br><br>
<div dir="ltr">
Hi Junaid,<br><br>Here is the output of the diffrent command executed on both server.<br>Command show identical output on both server<br><br># getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus<br>getfattr: Removing leading '/' from absolute path names<br>
# file: soft/venus<br>trusted.glusterfs.quota.size=0x0000000022ccac00<br><br> # getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/test<br>getfattr: Removing leading '/' from absolute path names<br># file: soft/venus/test<br>
trusted.glusterfs.quota.size=0x0000000000000000<br><br> # getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/psa <br>getfattr: Removing leading '/' from absolute path names<br># file: soft/venus/psa<br>
trusted.glusterfs.quota.size=0x000000000013d000<br><br> # getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/soft<br>getfattr: Removing leading '/' from absolute path names<br># file: soft/venus/soft<br>
trusted.glusterfs.quota.size=0x000000000d15ce00<br>
<br><br># gluster volume quota venus list<br> path limit_set size<br>----------------------------------------------------------------------------------<br>/ 2GB 556.8MB<br>
/test 100MB 0Bytes<br>/psa 200MB 1.2MB<br>/soft 1500MB 209.4MB<br><br><br>Indeed there is 90% of small file (~60000 files).<br>
<br>Thx<br><br>Anthony<br><br><br><div><hr>From: <a href="mailto:junaid@gluster.com" target="_blank">junaid@gluster.com</a><br>Date: Fri, 2 Sep 2011 06:18:30 +0530<br>Subject: Re: [Gluster-users] quota calculation gluster 3.2.3<br>
To: <a href="mailto:sokar6012@hotmail.com" target="_blank">sokar6012@hotmail.com</a><br>CC: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br><br>Hi Anthony,<div><br></div><div>To debug this further, can you send the output of </div>
<div><br></div><div>getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus</div><div>getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/test</div>
<div>getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/psa</div><div>getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/soft</div>
<div><br></div><div>from both the machine. Also check the output of</div><div><br></div><div>gluster volume quota <span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px">Replicate list</span></div>
<div><font face="arial, sans-serif"><span style="border-collapse:collapse"><br></span></font></div><div><font face="arial, sans-serif"><span style="border-collapse:collapse">Sometimes it takes a small amount of time to bring up the sizes. Also, what kind of data where you creating (I mean large amount of small files or large files because in case of small files, the directory sizes are not accounted by quota to calculate the size unlike the du -h command which uses the directory size as well).<br>
</span></font></div><div><font face="arial, sans-serif"><span style="border-collapse:collapse"><br></span></font></div><div><font face="arial, sans-serif"><span style="border-collapse:collapse">Junaid<br>
</span></font><div>On Thu, Sep 1, 2011 at 1:37 PM, anthony garnier <span dir="ltr"><<a href="mailto:sokar6012@hotmail.com" target="_blank">sokar6012@hotmail.com</a>></span> wrote:<br><blockquote style="padding-left:1ex">
<div><div dir="ltr">
Hi all,<br><br>I've enable quota but I'm a bit confused by values displayed by GlusterFS<br><br>Here is my volume : <br><br>Volume Name: venus<br>Type: Replicate<br>Status: Started<br>Number of Bricks: 2<br>Transport-type: tcp<br>
Bricks:<br>Brick1: ylal3020:/soft/venus<br>Brick2: ylal3030:/soft/venus<br>Options Reconfigured:<br>features.limit-usage: /test:100MB,/psa:200MB,/soft:1500MB,/:2GB<br>features.quota: on<br>diagnostics.client-log-level: ERROR<br>
diagnostics.brick-log-level: ERROR<br>network.ping-timeout: 10<br>performance.cache-size: 2GB<br>nfs.port: 2049<br><br>I've got 3 folders in the backend (/soft/venus) : <br>psa 160MB (with du -sh)<br>soft 1.2GB<br>
test 12KB<br>Total 1.4GB<br><br>But when I list the quota with gluster I got : <br># gluster volume quota venus list<br> path limit_set size<br>----------------------------------------------------------------------------------<br>
/test 100MB 12.0KB <= This one is OK<br>/psa 200MB 64.4MB <= not OK<br>/soft 1500MB 281.8MB <= not OK<br>
/ 2GB 346.2MB <= not OK<br><br>Any idea ?<br><br>Regards,<br><font color="#888888"><br>Anthony Garnier<br><br><br><br> </font></div></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
<br></blockquote></div><br></div></div> </div></div> </div></div></div></div> </div></div>
</blockquote></div><br></div></div></div></div> </div></div>
</blockquote></div><br></div></div></div></div> </div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>