<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
<br>
<blockquote
cite="mid:CAN6e=3MryzbSX0wi=GbSZxQDk_xUVSoocKx7JGFp92iauqkB6g@mail.gmail.com"
type="cite">
<div class="gmail_quote">On Wed, Sep 7, 2011 at 4:27 PM, Dan
Bretherton <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:d.a.bretherton@reading.ac.uk">d.a.bretherton@reading.ac.uk</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
<div>
<div class="h5"><br>
On 17/08/11 16:19, Dan Bretherton wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt
0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
<br>
<br>
<br>
Dan Bretherton wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt
0pt 0.8ex; border-left: 1px solid rgb(204, 204,
204); padding-left: 1ex;">
<br>
On 15/08/11 20:00, <a moz-do-not-send="true"
href="mailto:gluster-users-request@gluster.org"
target="_blank">gluster-users-request@gluster.org</a>
wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt
0pt 0pt 0.8ex; border-left: 1px solid rgb(204,
204, 204); padding-left: 1ex;">
Message: 1<br>
Date: Sun, 14 Aug 2011 23:24:46 +0300<br>
From: "Deyan Chepishev - SuperHosting.BG"<<a
moz-do-not-send="true"
href="mailto:dchepishev@superhosting.bg"
target="_blank">dchepishev@superhosting.bg</a>><br>
Subject: [Gluster-users] cluster.min-free-disk
separate for each<br>
brick<br>
To: <a moz-do-not-send="true"
href="mailto:gluster-users@gluster.org"
target="_blank">gluster-users@gluster.org</a><br>
Message-ID:<<a moz-do-not-send="true"
href="mailto:4E482F0E.3030604@superhosting.bg"
target="_blank">4E482F0E.3030604@superhosting.bg</a>><br>
Content-Type: text/plain; charset=UTF-8;
format=flowed<br>
<br>
Hello,<br>
<br>
I have a gluster set up with very different brick
sizes.<br>
<br>
brick1: 9T<br>
brick2: 9T<br>
brick3: 37T<br>
<br>
with this configuration if I set the parameter
cluster.min-free-disk to 10% it<br>
applies to all bricks which is quite uncomfortable
with these brick sizes,<br>
because 10% for the small bricks are ~ 1T but for
the big brick it is ~3.7T and<br>
what happens at the end is that if all brick go to
90% usage and I continue<br>
writing, the small ones eventually fill up to 100%
while the big one has enough<br>
free space.<br>
<br>
My question is, is there a way to set
cluster.min-free-disk per brick instead<br>
setting it for the entire volume or any other way
to work around this problem ?<br>
<br>
Thank you in advance<br>
<br>
Regards,<br>
Deyan<br>
<br>
</blockquote>
Hello Deyan,<br>
<br>
I have exactly the same problem and I have asked
about it before - see links below.<br>
<br>
<a moz-do-not-send="true"
href="http://community.gluster.org/q/in-version-3-1-4-how-can-i-set-the-minimum-amount-of-free-disk-space-on-the-bricks/"
target="_blank">http://community.gluster.org/q/in-version-3-1-4-how-can-i-set-the-minimum-amount-of-free-disk-space-on-the-bricks/</a>
<br>
<a moz-do-not-send="true"
href="http://gluster.org/pipermail/gluster-users/2011-May/007788.html"
target="_blank">http://gluster.org/pipermail/gluster-users/2011-May/007788.html</a><br>
<br>
My understanding is that the patch referred to in
Amar's reply in the May thread prevents a
"migrate-data" rebalance operation failing by
running out of space on smaller bricks, but that
doesn't solve the problem we are having. Being able
to set min-free-disk for each brick separately would
be useful, as would being able to set this value as
a number of bytes rather than a percentage.
However, even if these features were present we
would still have a problem when the amount of free
space becomes less than min-free-disk, because this
just results in a warning message in the logs and
doesn't actually prevent more files from being
written. In other words, min-free-disk is a soft
limit rather than a hard limit. When a volume is
more than 90% full there may still be hundreds of
gigabytes of free space spread over the large
bricks, but the small bricks may each only have a
few gigabytes left of even less. Users do "df" and
see lots of free space in the volume so they
continue writing files. However, when GlusterFS
chooses to write a file to a small brick, the write
fails with "device full" errors if the file grows
too large, which is often the case here with files
typically several gigabytes in size for some
applications.<br>
<br>
I would really like to know if there is a way to
make min-free-disk a hard limit. Ideally, GlusterFS
would chose a brick on which to write a file based
on how much free space it has left rather than
choosing a brick at random (or however it is done
now). That would solve the problem of non-uniform
brick sizes without the need for a hard
min-free-disk limit.<br>
<br>
Amar's comment in the May thread about QA testing
being done only on volumes with uniform brick sizes
prompted me to start standardising on a uniform
brick size for each volume in my cluster. My
impression is that implementing the features needed
for users with non-uniform brick sizes is not a
priority for Gluster, and that users are all
expected to use uniform brick sizes. I really think
this fact should be stated clearly in the GlusterFS
documentation, in the sections on creating volumes
in the Administration Guide for example. That would
stop other users from going down the path that I did
initially, which has given me a real headache
because I am now having to move tens of terabytes of
data off bricks that are larger than the new
standard size.<br>
<br>
Regards<br>
Dan.<br>
<br>
</blockquote>
Hello,<br>
<br>
This is really bad news, because I already migrated my
data and I just realized that I am screwed because
Gluster just does not care about the brick sizes.<br>
It is impossible to move to uniform brick sizes.<br>
<br>
Currently we use 2TB HDDs, but the disks are growing
and soon we will probably use 3TB hdds or whatever
other larges sizes appear on the market. So if we
choose to use raid5 and some level of redundancy (for
example 6hdds in raid5, no matter what their size is)
this sooner or later will lead us to non uniform
bricks which is a problem and it is not correct to
expect that we always can or want to provide uniform
size bricks.<br>
<br>
With this way of thinking if we currently have 10T
from 6x2T in hdd5, at some point when there is a 10T
on a single disk we will have to use no raid just
because gluster can not handle non uniform bricks.<br>
<br>
Regards,<br>
Deyan<br>
<br>
</blockquote>
<br>
I think Amar might have provided the answer in his
posting to the thread yesterday, which has just appeared
in my autospam folder.<br>
<br>
<a moz-do-not-send="true"
href="http://gluster.org/pipermail/gluster-users/2011-August/008579.html"
target="_blank">http://gluster.org/pipermail/gluster-users/2011-August/008579.html</a><br>
<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt
0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
With size option, you can have a hardbound on
min-free-disk<br>
</blockquote>
This means that you can set a hard limit on
min-free-disk, and set a value in GB that is bigger than
the biggest file that is ever likely to be written.
This looks likely to solve our problem and make
non-uniform brick sizes a practical proposition. I wish
I had known about this back in May when I embarked on my
cluster restructuring exercise; the issue was discussed
in this thread in May as well: <a
moz-do-not-send="true"
href="http://gluster.org/pipermail/gluster-users/2011-May/007794.html"
target="_blank">http://gluster.org/pipermail/gluster-users/2011-May/007794.html</a><br>
<br>
Once I have moved all the data off the large bricks and
standardised on a uniform brick size, it will be
relatively easy to stick to this because I use LVM. I
create logical volumes for new bricks when a volume
needs extending. The only problem with this approach is
what happens when the amount of free space left on a
server is less than the size of the brick you want to
create. The only option then would be to use new
servers, potentially wasting several TB of free space on
existing servers. The standard brick size for most of
my volumes is 3TB, which allows me to use a mixture of
small servers and large servers in a volume and limits
the amount of free space that would be wasted if there
wasn't quite enough free space on a server to create
another brick. Another consequence of having 3TB bricks
is that a single server typically has two more more
bricks belonging to a the same volume, although I do my
best to distribute the volumes across different servers
in order to spread the load. I am not aware of any
problems associated with exporting multiple bricks from
a single server and it has not caused me any problems so
far that I am aware of.<br>
<br>
-Dan.<br>
<br>
</blockquote>
</div>
</div>
Hello Deyan,<br>
<br>
Have you tried giving min-free-disk a value in gigabytes, and
if so does it prevent new files being written to your bricks
when they are nearly full? I recently tried it myself and
found that min-free-disk had no effect all. I deliberately
filled my test/backup volume and most of the bricks became 100
full. I set min-free-disk to "20GB", as reported in "gluster
volume ... info" below.<br>
<br>
cluster.min-free-disk: 20GB<br>
<br>
Unless I am doing something wrong it seems as though we can
not "have a hardbound on min-free-disk" after all, and uniform
brick size is therefore an essential requirement. It still
doesn't say that in the documentation, at least not in the
volume creation sections.
<div>
<div class="h5"><br>
<br>
-Dan.<br>
<br>
</div>
</div>
</blockquote>
</div>
On 08/09/11 06:35, Raghavendra Bhat wrote:<br>
> This is how it is supposed to work.<br>
><br>
> Suppose a distribute volume is created with 2 bricks. 1st
brick is having 25GB of free space, 2nd disk has 35 GB of free
space. If one sets a 30GB of minimum-free-disk through volume set
(gluster volume set <volname> min-free-disk 30GB), then
whenever files are created, if the file is hashed to the 1st brick
(which has 25GB of free space), then actual file will be created
in the 2nd brick to which a linkfile will be created in the 1st
brick. So the linkfile points to the actual file. A warning
message indicating minimum free disk limit has been crosses and
adding more nodes will be printed in the glusterfs log file. So
any file which is hashed to the 1st brick will be created in the
2nd brick.<br>
><br>
> Once the free space of 2nd brick also comes below 30 GB, then
the files will be created in the respective hashed bricks only.
There will be a warning message in the log file about the 2nd
brick also crossing the minimum free disk limit.<br>
><br>
> Regards,<br>
> Raghavendra Bhat<br>
<br>
</blockquote>
Dear Raghavendra,<br>
Thanks for explaining this to me. This mechanism should allow a
volume to function correctly with non-uniform brick sizes even
though min-free-disk is not a hard limit. I can understand now why
I had so many problems with the default value of 10% for
min-free-disk. 10% of a large brick can be very large compared to
10% of a small brick, so when they started filling up at the same
rate after all had less than 10% free space the small bricks usually
filled up long before large ones, giving "device full" errors even
when df still showed a lot of free space in the volume. At least
now we can minimise this effect by setting min-free-disk to a value
in GB.<br>
<br>
-Dan.<br>
<br>
</body>
</html>