<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
<blockquote cite="mid:4F015FF7.1000305@gluster.com" type="cite">On
Monday 02 January 2012 05:05 AM, Dan Bretherton wrote:
<br>
<blockquote type="cite">
<blockquote type="cite">On 03/10/11 19:08, Dan Bretherton wrote:
<br>
<blockquote type="cite">
<br>
On 02/10/11 02:12, Amar Tumballi wrote:
<br>
<blockquote type="cite">Dan,
<br>
<br>
Answer inline.
<br>
<br>
On 02-Oct-2011, at 1:26 AM, Dan
Bretherton<a class="moz-txt-link-rfc2396E" href="mailto:d.a.bretherton@reading.ac.uk"><d.a.bretherton@reading.ac.uk></a> wrote:
<br>
<br>
<blockquote type="cite">Hello All,
<br>
I have been testing rebalance...migrate-data in
GlusterFS version 3.2.3, following add-brick and
fix-layout. After migrate-data the the volume is 97%
full with some bricks being 100% full. I have not added
any files to the volume so there should be an amount of
free space at least as big as the new bricks that were
added. However, it seems as if all the extra space has
been taken up with files matching the pattern .*.gfs*.
I presume these are temporary files used for the
transfer real files, which should have been renamed once
the transfers were completed and verified, and the
original versions deleted. The new bricks contain
mostly these temporary files, and zero byte link files
pointing to the corresponding real files on other
bricks. An example of such a pair is shown below.
<br>
<br>
---------T 1 root root 0 Sep 30 03:14
/mnt/local/glusterfs/root/backup/behemoth_system/bin
<br>
-rwxr-xr-x 1 root root 60416 Sep 30 18:20
/mnt/local/glusterfs/root/backup/behemoth_system/bin/.df.gfs60416
<br>
<br>
Is this a known bug, and is there a work-around? If
not, is it safe to delete the .*.gfs* files so I can at
least use the volume?
<br>
<br>
</blockquote>
This is not a known issue but surely seems like a bug. If
the source file is intact you can delete the temp file to
get the space back. Also if md5sum is same, you can rename
temp file to original, so you get space in existing
bricks.
<br>
<br>
Regards,
<br>
Amar
<br>
<br>
<br>
<blockquote type="cite">Regards
<br>
Dan Bretherton
<br>
<br>
_______________________________________________
<br>
Gluster-users mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<br>
<a class="moz-txt-link-freetext" href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a>
<br>
</blockquote>
</blockquote>
Amar- Thanks for the information and the patch. The
etc-glusterd-mount-<volname>.log file can be
downloaded from here:
<br>
<br>
<a class="moz-txt-link-freetext" href="http://www.nerc-essc.ac.uk/~dab/etc-glusterd-mount-backup.log.tar.gz">http://www.nerc-essc.ac.uk/~dab/etc-glusterd-mount-backup.log.tar.gz</a>
<br>
<br>
I am using CentOS 5.5 by the way.
<br>
<br>
-Dan.
<br>
<br>
</blockquote>
Hello again-
<br>
I tested the patch and I confirm that it works; there are no
*.gfs* files in my volume after performing a migrate-data
operation. However there is still something not quite right.
One of the replicated brick pairs is 100% full, whereas the
others are approximately 50% full. I would have expected all
the bricks to contain roughly the same amount of data after
migrate-data, and this effect is mainly what I want to use
migrate-data for. Do you why this might have happened or how
to avoid it? The log files from the latest migrate-data
operation can be downloaded from here:
<br>
<br>
<a class="moz-txt-link-freetext" href="http://www.nerc-essc.ac.uk/~dab/backup_migrate-data_logs.tar.gz">http://www.nerc-essc.ac.uk/~dab/backup_migrate-data_logs.tar.gz</a>
<br>
<br>
-Dan.
<br>
</blockquote>
<br>
Hello Amar and gluster-users,
<br>
<br>
I have tested rebalance...migrate-data in version 3.2.5 and
found three serious problems still present.
<br>
<br>
1) There are lots of *.gfs* files after migrate-data. This
didn't happen when I tested the patched version of 3.2.4.
<br>
2) There are lots of duplicate files after migrate-data, i.e.
lots of files seen twice at the mount point. I have never seen
this happen before, and I would really like to know how to
repair the volume. There are ~6000 duplicates out of a total of
~1 million files in the volume, so dealing with each one
individually would be impractical.
<br>
3) A lot of files have wrong permissions after migrate-data.
For example, -rwsr-xr-x commonly becomes -rwxr-xr-x, and
-rw-rw-r-- commonly becomes -rw-r--r--.
<br>
<br>
Are these known problems, and if so is there a new version with
fixes in the pipeline?
<br>
<br>
Regards
<br>
Dan.
<br>
<br>
_______________________________________________
<br>
Gluster-users mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<br>
<a class="moz-txt-link-freetext" href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a>
<br>
</blockquote>
<br>
Hi Dan,
<br>
<br>
<br>
I tried the following steps
<br>
----------------------------
<br>
<br>
1. Created a replicate volume
<br>
2. filled the volume with 100000mu files
<br>
3. Added two more bricks with very less space to make sure out of
space condition occurs (now volume type is distributed-replicate).
<br>
4. After i start the rebalance, once the newly added bricks were
full, log messages were showing "out of disk space" messages, but
migration didn't happened.
<br>
<br>
Now on the mount point i could not see any *.gfs* files, and
permissions for these files were same even after rebalance.
<br>
<br>
<br>
Pleases let me know if i am missing something.
<br>
<br>
<br>
Thanks,
<br>
Shylesh
<br>
<br>
</blockquote>
Hello Shylesh- Thanks for trying to reproduce the problem. There
are clearly a lot of files in your test volume but you didn't say
how big they are or if any of them are symbolic links. The volume I
have been testing migrate-data on is 3.6TB in size and is 64% full.
There are 22 bricks on 10 servers. There are ~45000 symlinks and
~1 million files, varying in size from vary small (~KB) to very
large (several GB). It contains, among other things, an operating
system backup (where a lot of the symlinks and small files come
from) and ~1.5TB of large NetCDF
(<a class="moz-txt-link-freetext" href="http://www.unidata.ucar.edu/software/netcdf/">http://www.unidata.ucar.edu/software/netcdf/</a>) files. All the
servers have bricks belonging to other volumes as well as the volume
being tested. I don't know if any of these volume or file
characteristics are responsible for the problems that occurred
during the migrate-data process. It would be relatively easy to
have several servers involved in your test volume and several bricks
per server. Perhaps you could transfer some Linux system files onto
it to recreate some of the features of my test volume, and it would
also be worth having a range of files sizes up to ~8GB, like the
NetCDF files I referred to.<br>
<br>
The volume mount log file on the server that carried out the
migrate-data operation contains a lot of errors like the following.<br>
<blockquote>[2011-12-31 11:11:44.621287] E
[client3_1-fops.c:2056:client3_1_link_cbk] 0-backup-client-17:
remote operation failed: File exists<br>
[2011-12-31 11:11:44.621330] E
[client3_1-fops.c:2056:client3_1_link_cbk] 0-backup-client-16:
remote operation failed: File exists<br>
[2011-12-31 11:11:44.623785] W
[fuse-bridge.c:1354:fuse_rename_cbk] 0-glusterfs-fuse: 46317596:
/users/dab/backup/previous_version/data/gorgon/users/dab/Installations/glibc-2.5/sysdeps/unix/sysv/linux/s390/s390-32/.versionsort64.c.gfs56
->
/users/dab/backup/previous_version/data/gorgon/users/dab/Installations/glibc-2.5/sysdeps/unix/sysv/linux/s390/s390-32/versionsort64.c
=> -1 (File exists)<br>
</blockquote>
The brick log files contain some errors relating to *.gfs* files,
such as the following.<br>
<blockquote>[2011-12-31 11:27:26.348954] I
[server3_1-fops.c:1050:server_link_cbk] 0-backup-server: 9546975:
LINK /users/dab/backup/previous_version/data/gorgon/users/dab/Installations/apache-ant-1.7.0/docs/antlibs/.index.html.gfs7312
(-3867887006) ==> -1 (File exists)<br>
[2011-12-31 11:27:26.917764] E [posix.c:2213:posix_link]
0-backup-posix: link
/users/dab/backup/previous_version/data/gorgon/users/dab/Installations/apache-ant-1.7.0/bin/.antRun.pl.gfs2199
to
/users/dab/backup/previous_version/data/gorgon/users/dab/Installations/apache-ant-1.7.0/bin/antRun.pl
failed:File exists<br>
[2011-12-31 11:27:26.917796] I
[server3_1-fops.c:1050:server_link_cbk] 0-backup-server: 9547232:
LINK /users/dab/backup/previous_version/data/gorgon/users/dab/Installations/apache-ant-1.7.0/bin/.antRun.pl.gfs2199
(-2737554403) ==> -1 (File exists)<br>
<br>
[2011-12-31 12:08:28.88889] E [posix.c:921:posix_setattr]
0-backup-posix: setattr (lstat) on
/mnt/local/glusterfs/users/dab/backup/data/gorgon/users/dab/SGE/src/gridengine/source/dist/mpi/myrinet/.gmps.gfs3139
failed: No such file or directory<br>
[2011-12-31 12:08:28.170369] I
[server3_1-fops.c:1526:server_setattr_cbk] 0-backup-server:
5421009: SETATTR
/users/dab/backup/data/gorgon/users/dab/SGE/src/gridengine/source/dist/mpi/myrinet/.gmps.gfs3139
(-783637172) ==> -1 (No such file or directory)<br>
</blockquote>
The log files can be downloaded as files bdan0.tar.gz,
bdan12.tar.gz, bdan13.tar.gz, bdan14.tar.gz, bdan1.tar.gz,
bdan2.tar.gz, bdan3.tar.gz, bdan6.tar.gz, bdan7.tar.gz
and perseus.tar.gz from <a class="moz-txt-link-freetext" href="http://www.nerc-essc.ac.uk/~dab">http://www.nerc-essc.ac.uk/~dab</a>.<br>
<br>
The migrate-data operation was carried out on a server called
perseus. Here is the volume information so you can make sense of
the logs.<br>
[root@bdan12 ~]# gluster volume info backup<br>
<br>
Volume Name: backup<br>
Type: Distributed-Replicate<br>
Status: Started<br>
Number of Bricks: 11 x 2 = 22<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: bdan0.nerc-essc.ac.uk:/mnt/local/glusterfs<br>
Brick2: bdan1.nerc-essc.ac.uk:/mnt/local/glusterfs<br>
Brick3: bdan2.nerc-essc.ac.uk:/mnt/local/glusterfs<br>
Brick4: bdan3.nerc-essc.ac.uk:/mnt/local/glusterfs<br>
Brick5: bdan6.nerc-essc.ac.uk:/mnt/local/glusterfs<br>
Brick6: bdan7.nerc-essc.ac.uk:/mnt/local/glusterfs<br>
Brick7: perseus.nerc-essc.ac.uk:/mnt/local/glusterfs<br>
Brick8: perseus.nerc-essc.ac.uk:/mnt/local2/glusterfs<br>
Brick9: perseus.nerc-essc.ac.uk:/backup/glusterfs<br>
Brick10: bdan14.nerc-essc.ac.uk:/backup/glusterfs<br>
Brick11: perseus.nerc-essc.ac.uk:/backup2/glusterfs<br>
Brick12: bdan14.nerc-essc.ac.uk:/backup2/glusterfs<br>
Brick13: bdan12.nerc-essc.ac.uk:/backup2/glusterfs<br>
Brick14: bdan13.nerc-essc.ac.uk:/backup2/glusterfs<br>
Brick15: bdan12.nerc-essc.ac.uk:/backup3/glusterfs<br>
Brick16: bdan13.nerc-essc.ac.uk:/backup3/glusterfs<br>
Brick17: bdan12.nerc-essc.ac.uk:/backup/glusterfs<br>
Brick18: bdan13.nerc-essc.ac.uk:/backup/glusterfs<br>
Brick19: perseus.nerc-essc.ac.uk:/backup3/glusterfs<br>
Brick20: bdan14.nerc-essc.ac.uk:/backup3/glusterfs<br>
Brick21: perseus.nerc-essc.ac.uk:/backup4/glusterfs<br>
Brick22: bdan14.nerc-essc.ac.uk:/backup4/glusterfs<br>
Options Reconfigured:<br>
performance.quick-read: off<br>
performance.cache-refresh-timeout: 0<br>
performance.stat-prefetch: off<br>
cluster.min-free-disk: 34GB<br>
nfs.disable: on<br>
<br>
Regards<br>
Dan.<br>
<br>
<br>
<br>
</body>
</html>