<div dir="ltr">Hi all,<div><br></div><div>I have 3 peers (peer-A, peer-B and peer-C). I tried to use add-brick and remove-brick to replace peers.</div><div>(version: glusterfs 3.4)</div><div><br></div><div>What I did:</div>
<div><ol><li>created a distribute volume with two 10-GB bricks (peer-A:/brick and peer-B:/brick. Actually they are 9.7 GB after ext4 formatting).</li><li>mount it and write 16 1-GB files in to it (command: seq 16 | xargs -i dd if=/dev/zero of=/mnt/file-{} bs=1G count=1).</li>
<li>add peer-C:/brick (also 10-GB) to this volume.</li><li>execute remove peer-A:/brick start.</li><li>check remove status and wait until all of the hosts are completed.</li><li>execute remove peer-A:/brick commit.</li></ol>
After step 6, I lost 2 files in the volume.</div><div><br></div><div><br></div><div>I list the files in bricks after step 2 and step 5:</div><div>After step 2:<br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div>peer-A:/brick:</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-1</div></div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-12</div>
</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-14</div></div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-15</div></div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:08 file-16</div>
</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-3</div></div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-6</div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div><br></div><div>peer-B:/brick:</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-10</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-11</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-13</div>
<div>---------T 2 root root 0 Mar 4 17:07 file-15</div><div>---------T 2 root root 0 Mar 4 17:07 file-16</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-2</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-4</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-5</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-7</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-8</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-9</div></div><div><br></div></blockquote><div>After step 5:<br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div>peer-A:/brick:</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-15</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:08 file-16</div></div><div><br></div><div>peer-B:/brick:</div>
<div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-10</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-11</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-13</div>
<div>---------T 2 root root 1073741824 Mar 4 17:17 file-15</div><div>---------T 2 root root 1073741824 Mar 4 17:17 file-16</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-2</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-4</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-5</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-7</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-8</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-9</div></div><div><br></div><div>peer-C:/brick:</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-1</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-12</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-14</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-3</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-6</div></div></blockquote><div><br></div><div>After step 6, I lost file-15 and file-16 in the volume. </div><div>Anyone know why file-15 and file-16 are not moved to peer-C?</div>
<div>If it's caused by peer-B is full, why does the status show "completed"?</div><div><br></div><div><div>Node Rebalanced-files size scanned failures skipped status run-time in secs</div>
<div>--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</div><div>localhost 5 5.0GB 21 0 completed 126.00</div>
<div>localhost 5 5.0GB 21 0 completed 126.00</div><div>localhost 5 5.0GB 21 0 completed 126.00</div>
<div>localhost 5 5.0GB 21 0 completed 126.00</div></div><div><br></div><div><br></div><div>--</div><div>Best regards,</div><div>Johnny</div><div><a href="mailto:j1899j1899@gmail.com">j1899j1899@gmail.com</a></div>
<div><br></div></div>