<div dir="ltr"><span style="font-family:arial,sans-serif;font-size:18px">Hi all,</span><div style="font-family:arial,sans-serif;font-size:18px"><br></div><div style="font-family:arial,sans-serif;font-size:18px">I have 3 peers (peer-A, peer-B and peer-C). I tried to use add-brick and remove-brick to replace peers.</div>
<div style="font-family:arial,sans-serif;font-size:18px">(version: glusterfs 3.4)</div><div style="font-family:arial,sans-serif;font-size:18px"><br></div><div style="font-family:arial,sans-serif;font-size:18px">What I did:</div>
<div style="font-family:arial,sans-serif;font-size:18px"><ol><li style="margin-left:15px">created a distribute volume with two 10-GB bricks (peer-A:/brick and peer-B:/brick. Actually they are 9.7 GB after ext4 formatting).</li>
<li style="margin-left:15px">mount it and write 16 1-GB files in to it (command: seq 16 | xargs -i dd if=/dev/zero of=/mnt/file-{} bs=1G count=1).</li><li style="margin-left:15px">add peer-C:/brick (also 10-GB) to this volume.</li>
<li style="margin-left:15px">execute remove peer-A:/brick start.</li><li style="margin-left:15px">check remove status and wait until all of the hosts are completed.</li><li style="margin-left:15px">execute remove peer-A:/brick commit.</li>
</ol>After step 6, I lost 2 files in the volume.</div><div style="font-family:arial,sans-serif;font-size:18px"><br></div><div style="font-family:arial,sans-serif;font-size:18px"><br></div><div style="font-family:arial,sans-serif;font-size:18px">
I list the files in bricks after step 2 and step 5:</div><div style="font-family:arial,sans-serif;font-size:18px">After step 2:<br></div><blockquote style="font-family:arial,sans-serif;font-size:18px;margin:0px 0px 0px 40px;border:none;padding:0px">
peer-A:/brick:</blockquote><blockquote style="font-family:arial,sans-serif;font-size:18px;margin:0px 0px 0px 40px;border:none;padding:0px"><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-1</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-12</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-14</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-15</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:08 file-16</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-3</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-6</div></blockquote><blockquote style="font-family:arial,sans-serif;font-size:18px;margin:0px 0px 0px 40px;border:none;padding:0px">
<div><br></div><div>peer-B:/brick:</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-10</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-11</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-13</div>
<div>---------T 2 root root 0 Mar 4 17:07 file-15</div><div>---------T 2 root root 0 Mar 4 17:07 file-16</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-2</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-4</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-5</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-7</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-8</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-9</div></div><div><br></div></blockquote><div style="font-family:arial,sans-serif;font-size:18px">
After step 5:<br></div><blockquote style="font-family:arial,sans-serif;font-size:18px;margin:0px 0px 0px 40px;border:none;padding:0px"><div>peer-A:/brick:</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-15</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:08 file-16</div></div><div><br></div><div>peer-B:/brick:</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-10</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-11</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-13</div><div>---------T 2 root root 1073741824 Mar 4 17:17 file-15</div><div>---------T 2 root root 1073741824 Mar 4 17:17 file-16</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-2</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-4</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-5</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-7</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-8</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-9</div>
</div><div><br></div><div>peer-C:/brick:</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-1</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-12</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-14</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-3</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-6</div></div></blockquote><div style="font-family:arial,sans-serif;font-size:18px">
<br></div><div style="font-family:arial,sans-serif;font-size:18px">After step 6, I can't find file-15 and file-16 in the volume.</div><div style="font-family:arial,sans-serif;font-size:18px">Anyone know why file-15 and file-16 are not moved to peer-C?</div>
<div style="font-family:arial,sans-serif;font-size:18px">If it's caused by peer-B is full, why does the status show "completed"?</div><div style="font-family:arial,sans-serif;font-size:18px"><br></div><div style="font-family:arial,sans-serif;font-size:18px">
<div>Node Rebalanced-files size scanned failures skipped status run-time in secs</div><div>--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</div>
<div>localhost 5 5.0GB 21 0 completed 126.00</div><div>localhost 5 5.0GB 21 0 completed 126.00</div>
<div>localhost 5 5.0GB 21 0 completed 126.00</div><div>localhost 5 5.0GB 21 0 completed 126.00</div>
</div><div style="font-family:arial,sans-serif;font-size:18px"><br></div><div style="font-family:arial,sans-serif;font-size:18px"><br></div><div style="font-family:arial,sans-serif;font-size:18px">--</div><div style="font-family:arial,sans-serif;font-size:18px">
Best regards,</div><div style="font-family:arial,sans-serif;font-size:18px">Johnny</div></div>