<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
Hello Shylesh,<br>
Thanks for looking into this for me. I think the ext4 features are
missing because the filesystems were accidentally formatted as ext3
and then mounted as ext4. I didn't realise that was possible until
I started investigating this fix-layout problem. I don't know how I
managed to make the same mistake on both replicated bricks but I
can't think of any other explanation. I mounted the filesystems as
ext3 and tried the rebalance again, but the result was the same.
Then I tried converting the filesystems to ext4, as described in
various CentOS forums and blogs including this one:
<a class="moz-txt-link-freetext" href="http://blog.secaserver.com/2011/08/linux-converting-ext3-ext4-for-centos-5">http://blog.secaserver.com/2011/08/linux-converting-ext3-ext4-for-centos-5</a>.
Unfortunately the "Operation not supported" errors were still there
during the fix-layout, so it seems that the damage has already been
done by mounting the ext3 filesystems as ext4. Perhaps xattrs on
new files would be created correctly in the converted bricks, but I
really need to find a way to repair the GlusterFS xattrs on the
existing files. Is there a way of doing this?<br>
<br>
Regards<br>
Dan.
<blockquote cite="mid:4EFD6D2F.8050102@gluster.com" type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<title></title>
</blockquote>
<blockquote cite="mid:4EFD6D2F.8050102@gluster.com" type="cite">Hi
Dan,<br>
<br>
I created two bricks both have ext4 file system.<br>
<br>
The issue seems to be in fs features that you have disabled.<br>
<br>
Formatted the <b>brick1</b> with ext4:<br>
<br>
root@SERVER1 mnt]# dumpe2fs /dev/sda| grep 'Filesystem features'<br>
dumpe2fs 1.41.12 (17-May-2010)<br>
Filesystem features: has_journal ext_attr resize_inode
dir_index filetype needs_recovery extent flex_bg sparse_super
large_file huge_file uninit_bg dir_nlink extra_isize<br>
<br>
Formatted <b>brick 2</b> with ext4:<br>
[root@SERVER2 ~]# dumpe2fs /dev/sda| grep 'Filesystem features'<br>
dumpe2fs 1.41.12 (17-May-2010)<br>
Filesystem features: has_journal ext_attr resize_inode
dir_index filetype extent flex_bg sparse_super large_file<br>
<br>
As you said i have disabled some of the features from <b>brick2</b>.<br>
<br>
I created a distribute volume with these two bricks. created some
files on the mount point and tried setting xattr for these files.<br>
<br>
I got error messages<br>
=======================================================================================<br>
[2011-12-30 01:57:22.551634] I
[client3_1-fops.c:818:client3_1_setxattr_cbk] 1-test-client-1:
remote operation failed: Operation not supported<br>
[2011-12-30 01:57:22.551658] W [fuse-bridge.c:850:fuse_err_cbk]
0-glusterfs-fuse: 201305: SETXATTR() /92 => -1 (Operation not
supported)<br>
[2011-12-30 01:57:22.556490] I
[client3_1-fops.c:818:client3_1_setxattr_cbk] 1-test-client-1:
remote operation failed: Operation not supported<br>
[2011-12-30 01:57:22.556520] W [fuse-bridge.c:850:fuse_err_cbk]
0-glusterfs-fuse: 201311: SETXATTR() /95 => -1 (Operation not
supported)<br>
[2011-12-30 01:57:22.564089] I
[client3_1-fops.c:818:client3_1_setxattr_cbk] 1-test-client-1:
remote operation failed: Operation not supported<br>
[2011-12-30 01:57:22.564114] W [fuse-bridge.c:850:fuse_err_cbk]
0-glusterfs-fuse: 201321: SETXATTR() /100 => -1 (Operation not
supported)<br>
========================================================================================<br>
<br>
where as i created another volume with only <b>brick1</b> and
everything went smoothly. <br>
so i suspect problem is not with rebalance but with ext4 features
that are disabled on <b>brick2</b>.<br>
<br>
Please let me know if i am missing anything that can be tried.<br>
<br>
<br>
<br>
<br>
Thanks,<br>
Shylesh<br>
<br>
<blockquote
cite="mid:756EA019FD2A1648B9F14B0C16B440E8FC601D@mbx024-e1-nj-8.exch024.domain.local"
type="cite">
<div style="direction: ltr; font-family: Tahoma; color: rgb(0,
0, 0); font-size: 10pt;">
<div style="font-family: Times New Roman; color: rgb(0, 0, 0);
font-size: 16px;">
<hr tabindex="-1">
<div style="direction: ltr;" id="divRpF787186"><font
color="#000000" face="Tahoma" size="2"><b>From:</b> <a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:gluster-users-bounces@gluster.org">gluster-users-bounces@gluster.org</a>
[<a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:gluster-users-bounces@gluster.org">gluster-users-bounces@gluster.org</a>]
on behalf of Dan Bretherton [<a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:d.a.bretherton@reading.ac.uk">d.a.bretherton@reading.ac.uk</a>]<br>
<b>Sent:</b> Thursday, December 29, 2011 6:05 AM<br>
<b>To:</b> gluster-users<br>
<b>Subject:</b> [Gluster-users] fix-layout stalls with
xattr errors<br>
</font><br>
</div>
<div>Hello All-<br>
I am having problems with rebalance ... fix-layout in
version 3.2.5. I extended a volume with add-brick but the
fix-layout stalls after a small number of layout fixes and
does not make any more progress. I have tried the
operation twice on different servers with the same result.
The following errors are found in the fuse mount log file
on the server carrying out the operation.<br>
<br>
<blockquote>[2011-12-28 21:38:14.840013] I
[afr-common.c:1038:afr_launch_self_heal]
0-nemo2-replicate-4: background data self-heal
triggered. path:
/users/hzu/DATA/ERAINT/ORCA025/2010/snow_ERAINT_2010.nc<br>
[2011-12-28 21:38:15.93079] E
[client3_1-fops.c:1498:client3_1_fxattrop_cbk]
0-nemo2-client-8: remote operation failed: Operation not
supported<br>
[2011-12-28 21:38:15.93141] E
[client3_1-fops.c:1498:client3_1_fxattrop_cbk]
0-nemo2-client-9: remote operation failed: Operation not
supported<br>
[2011-12-28 21:38:15.93385] I
[client3_1-fops.c:1187:client3_1_fstat_cbk]
0-nemo2-client-8: remote operation failed: Operation not
supported<br>
[2011-12-28 21:38:15.93521] I
[client3_1-fops.c:1187:client3_1_fstat_cbk]
0-nemo2-client-9: remote operation failed: Operation not
supported<br>
</blockquote>
<br>
The file in the error message is a link, and it is not
broken as seen from the volume mount point or the bricks.<br>
<br>
There are some worrying error messages in the brick log
files for nemo2-client-8 and nemo2-client-9. Here are
some exerpts from the nemo2-client-8 log, which is similar
to the 0-nemo2-client-9 log.<br>
<br>
<blockquote>[2011-12-28 21:23:05.827877] W
[posix.c:3928:do_xattrop] 0-nemo2-posix: Extended
attributes not supported by filesystem<br>
[2011-12-28 21:23:05.827932] I
[server3_1-fops.c:1705:server_fxattrop_cbk]
0-nemo2-server: 8438: FXATTROP 0 (-2111276040) ==> -1
(Operation not support<br>
ed)<br>
[2011-12-28 21:23:05.828848] E
[posix.c:4200:posix_fstat] 0-nemo2-posix: fstat failed
on fd=0x2aaaac703804: Operation not supported<br>
[2011-12-28 21:23:05.828879] I
[server3_1-fops.c:1113:server_fstat_cbk] 0-nemo2-server:
8439: FSTAT 0 (-2111276040) ==> -1 (Operation not
supported)<br>
[2011-12-28 21:29:29.871213] W
[socket.c:1494:__socket_proto_state_machine]
0-tcp.nemo2-server: reading from socket failed. Error
(Transport endpoint i<br>
s not connected), peer (192.171.166.81:1003)<br>
[2011-12-28 21:29:29.871305] I
[server-helpers.c:360:do_lock_table_cleanup]
0-nemo2-server: inodelk released on
/users/hzu/DATA/ERAINT/ORCA025/2010/sno<br>
w_ERAINT_2010.nc<br>
[2011-12-28 21:29:29.871345] I
[server-helpers.c:485:do_fd_cleanup] 0-nemo2-server: fd
cleanup on
/users/hzu/DATA/ERAINT/ORCA025/2010/snow_ERAINT_2010.<br>
nc<br>
<br>
[2011-12-28 21:34:36.190023] I
[server-helpers.c:485:do_fd_cleanup] 0-nemo2-server: fd
cleanup on /<br>
[2011-12-28 21:34:36.190055] I
[server-helpers.c:485:do_fd_cleanup] 0-nemo2-server: fd
cleanup on /users<br>
[2011-12-28 21:34:36.190086] I
[server-helpers.c:485:do_fd_cleanup] 0-nemo2-server: fd
cleanup on /users/hzu<br>
[2011-12-28 21:34:36.190102] I
[server-helpers.c:485:do_fd_cleanup] 0-nemo2-server: fd
cleanup on /users/hzu/DATA<br>
[2011-12-28 21:34:36.190135] I
[server-helpers.c:485:do_fd_cleanup] 0-nemo2-server: fd
cleanup on /users/hzu/DATA/ERAINT<br>
[2011-12-28 21:34:36.190154] I
[server-helpers.c:485:do_fd_cleanup] 0-nemo2-server: fd
cleanup on /users/hzu/DATA/ERAINT/ORCA025<br>
[2011-12-28 21:34:36.190171] I
[server-helpers.c:485:do_fd_cleanup] 0-nemo2-server: fd
cleanup on /users/hzu/DATA/ERAINT/ORCA025/2009<br>
<br>
[2011-12-28 21:38:15.92433] I
[server3_1-fops.c:1705:server_fxattrop_cbk]
0-nemo2-server: 12228: FXATTROP 7 (-2111276040) ==>
-1 (Operation not supported)<br>
[2011-12-28 21:38:15.92743] E [posix.c:4200:posix_fstat]
0-nemo2-posix: fstat failed on fd=0x2aaaac703804:
Operation not supported<br>
[2011-12-28 21:38:15.92775] I
[server3_1-fops.c:1113:server_fstat_cbk] 0-nemo2-server:
12229: FSTAT 7 (-2111276040) ==> -1 (Operation not
supported)<br>
</blockquote>
<br>
The backend filesystems are ext4 and the are mounted with
options "acl,user_xattr". I tested extended attribute
support (as suggested here: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://gluster.org/pipermail/gluster-users/2010-December/006257.html"
target="_blank">
http://gluster.org/pipermail/gluster-users/2010-December/006257.html</a>)
and could not find any problems, so I don't understand the
"Extended attributes not supported by filesystem" error.
The only unusual thing about the filesystems is the
reduced number of filesystem features enabled compared to
other bricks. These are the ext4 features enabled.<br>
<br>
has_journal ext_attr resize_inode dir_index filetype
needs_recovery sparse_super large_file<br>
<br>
All the other bricks in the volume have these features
plus extent, flex_bg, huge_file, uninit_bg, dir_nlink and
extra_isize. I don't know if any of these missing ext4
features are part of the problem. Does anybody know
what's going on here?<br>
<br>
Regards<br>
Dan.<br>
<br>
<br>
</div>
</div>
</div>
</blockquote>
<br>
</blockquote>
</body>
</html>