<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:2 7 4 9 2 2 5 2 4 4;}
@font-face
{font-family:Courier;
panose-1:2 7 4 9 2 2 5 2 4 4;}
@font-face
{font-family:Cambria;
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:677779762;
mso-list-type:hybrid;
mso-list-template-ids:212480888 -1654883058 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"\(%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1
{mso-list-id:1549999676;
mso-list-type:hybrid;
mso-list-template-ids:1368194820 -1208855804 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
{mso-level-text:"\(%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:.25in;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>There is a post some time ago about migrating data using “remove-brick”.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><a href="http://www.gluster.org/pipermail/gluster-users/2012-October/034473.html">http://www.gluster.org/pipermail/gluster-users/2012-October/034473.html</a><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Is that approach reliable? Why is it still not officially documented? <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I followed the instruction to run the command on a distributed volume bricks, using gluster version 3.4.2. It works for small number of files. However, when I tried with a large storage space with a large number of files, after the migration, it looks like some files are missing.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Here it is the process:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoListParagraph style='margin-left:.25in;text-indent:-.25in;mso-list:l1 level1 lfo2;text-autospace:none'><![if !supportLists]><span style='font-family:"Arial","sans-serif"'><span style='mso-list:Ignore'>(1)<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Arial","sans-serif"'>Check volume info<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'># gluster volume info<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>Volume Name: content<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>Type: Distribute<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>Volume ID: 48485504-e89d-4146-afd3-356397e5d541<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>Status: Started<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>Number of Bricks: 3<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>Transport-type: tcp<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>Bricks:<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>Brick1: adsfsvip:/data/c01<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>Brick2: adsfsvip:/data/c03<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><b><span style='font-family:"Arial","sans-serif"'>Brick3: adsfsvip:/data/c02</span></b><span style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>Options Reconfigured:<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>nfs.disable: OFF<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>(2) Check number of files in the volume, for verification of content migration and rebalance.<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'># for i in 1 2 3; do tree /data/c0$i | grep files; done<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>661 directories, 27126 files<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>661 directories, 15619 files<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>661 directories, 28531 files<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'># tree /data | grep files<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>1995 directories, <b>71276 </b>files<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>(3) Running command "remove-brick" with "start" to start migrating content. Suppose we want to migrate content on Brick3 to other bricks.<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'># gluster volume remove-brick content adsfsvip:/data/c02 start<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>volume remove-brick start: success<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>ID: 06891565-b584-4ab5-a015-3bda991f2454<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>(4) The data migration may take some time, run the following command to periodically check the migration process.<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'>[root@ads1 glusterfs]# while [ 1 ] ; do gluster volume remove-brick content adsfsvip:/data/c02 status; echo; sleep 2 ; done<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> Node Rebalanced-files size scanned failures skipped status run-time in secs<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> localhost 4821 19.0GB 13427 0 in progress 286.00<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> Node Rebalanced-files size scanned failures skipped status run-time in secs<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> localhost 4849 19.1GB 13455 0 in progress 288.00<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> Node Rebalanced-files size scanned failures skipped status run-time in secs<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> localhost 4877 19.3GB 13656 0 <span style='color:green'>in progress</span> 290.00<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> ...</span><span style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> Node Rebalanced-files size scanned failures skipped status run-time in secs<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> localhost 24864 148.3GB 61459 0 in progress 1525.00<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> Node Rebalanced-files size scanned failures skipped status run-time in secs<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:Courier'> localhost 24871 148.3GB 61465 0 completed 1526.00<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>(5) Check the number of files at each brick in the volume. The number of files at the brick we just ran "remove-brick" over should be 0.<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>[root@ads1 glusterfs]# for i in 1 2 3; do tree /data/c0$i | grep files; done<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>661 directories, 29533 files<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>661 directories, <b>0 files</b><o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>661 directories, 35666 files<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>(6) However, the total number of files is reduced from <b>71276 to </b><span style='color:red'>65199. </span>Do we lose some files in the migration process?<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>[root@ads1 glusterfs]# tree /data | grep files<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Arial","sans-serif"'>1995 directories, <span style='color:red'>65199 files</span><o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thank you and best regards,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-family:"Cambria","serif";color:#1F497D'>Leo<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>