<div dir="ltr"><div><div><div><div><div><div><div><div><div>Dear Amar and glusterfs experts,<br><br></div>We use glusterfs 3.3.<br><br></div>xfs related debug info can be found in the following two thread.<br><a href="http://oss.sgi.com/archives/xfs/2013-04/msg00327.html">http://oss.sgi.com/archives/xfs/2013-04/msg00327.html</a><br>
<a href="http://www.spinics.net/lists/xfs/msg17893.html">http://www.spinics.net/lists/xfs/msg17893.html</a><br><br></div>The xfs shutdown due to potential racing unlink in xfs kernel on the following files:<br>directory ./glusterfs/indices/<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:20px;color:rgb(51,51,51);text-transform:none;font-size:14px;white-space:pre-wrap;font-family:Consolas,Courier,Helvetica,Arial;word-spacing:0px">xattrop<br>
</span></div><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:20px;color:rgb(51,51,51);text-transform:none;font-size:14px;white-space:pre-wrap;font-family:Consolas,Courier,Helvetica,Arial;word-spacing:0px">and files under this directory<br>
</span></div><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:20px;color:rgb(51,51,51);text-transform:none;font-size:14px;white-space:pre-wrap;font-family:Consolas,Courier,Helvetica,Arial;word-spacing:0px">The racing cause xfs unlinked list memory corrupt and then xfs shutdown.<br>
</span></div><div><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:20px;color:rgb(51,51,51);text-transform:none;font-size:14px;white-space:pre-wrap;font-family:Consolas,Courier,Helvetica,Arial;word-spacing:0px">You can see the significant </span>heavy remove/link pattern on a single<br>
inode int the following kernel trace.<br><a href="https://docs.google.com/file/d/0B7n2C4T5tfNCTlZGUVpnZENrZ3M/edit?usp=sharing" target="_blank">https://docs.google.com/file/d/0B7n2C4T5tfNCTlZGUVpnZENrZ3M/edit?usp=sharing</a><br>
</div><div>finally<br><...>-6908 [001] 8739.967623: xfs_iunlink: dev 8:16 ino 0x83a8b mode<br>
0100000, flags 0x0<br>
<...>-6909 [001] 8739.970252: xfs_iunlink: dev 8:16 ino 0x83a8b mode<br>
0100000, flags 0x0<br>
<br>
0x83a8b and 0x1b20b both hash to unlinked list bucket 11.<br></div><div>the shutdown then happens because of corrupt unlinked list.<br></div><div><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:20px;color:rgb(51,51,51);text-transform:none;font-size:14px;white-space:pre-wrap;font-family:Consolas,Courier,Helvetica,Arial;word-spacing:0px"><br>
</span></div><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:20px;color:rgb(51,51,51);text-transform:none;font-size:14px;white-space:pre-wrap;font-family:Consolas,Courier,Helvetica,Arial;word-spacing:0px">Now we move the </span>indices directory out of xfs volume to an ext4 volume and the crash not happens again.<br>
</div>This issue always happen when we rebalance glusterfs volumes(for example to growth the cluster).<br>The worst case ever happened is one day 8 of our servers xfs shutdown at the same time.<br></div>The xfs experts is now working on this and before there're any progresses I also want to ask help and sugesstions from you.<br>
</div>Thank you very much.<br><div><div><div><br><div><div><div><div><br></div></div></div></div></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/4/22 Amar Tumballi <span dir="ltr"><<a href="mailto:amarts@redhat.com" target="_blank">amarts@redhat.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 04/22/2013 08:58 AM, ·ûÓÀÌÎ wrote:<br>
> Dear Amar,<br>
> Sorry to bother but I want to ask some help related to glusterfs.<br>
> Recently we have encountered a xfs shutdown issue related to glusterfs<br>
> rebalance. Finally we found it is related to fops related to<br>
> ./glusterfs/indices/xattrop directory.<br>
> I find it is related to glusterfs features/index xlator.<br>
> My question is what does this xlator do? I find the indices dir can be<br>
> configured in server-side volume files. Is it safe to move is somewhere<br>
> else for online servers?<br>
><br>
<br>
</div>features/index translator keeps track of files which needs self-heal<br>
(with respect to replicate xlator). About changing the volume files (ie,<br>
moving the xlator position in volfile directly), we don't recommend it<br>
as the default setup is the only one which gets tested both by QE and<br>
almost all the user base.<br>
<br>
Anyways, couple of questions to you on the issue you are facing.<br>
<br>
* Please specify the version of GlusterFS you are using.<br>
* when you say we have encountered a xfs shutdown [1] related "glusterfs<br>
rebalance", can you post glusterfs rebalance logs?<br>
<br>
Regards,<br>
Amar<br>
<div class="HOEnZb"><div class="h5"><br>
> Thank you very much.<br>
><br>
> BTW, here's the xfs shutdown issue caused by glusterfs rebalance.<br>
> <a href="http://www.marshut.com/irunt/xfs-iunlink-remove-xfs-inotobp-returned-error-22-debugging.html" target="_blank">http://www.marshut.com/irunt/xfs-iunlink-remove-xfs-inotobp-returned-error-22-debugging.html</a><br>
><br>
><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>·ûÓÀÌÎ
</div>