<div dir="ltr">Hi all,<div><br></div><div>I found the root cause!</div><div>In master branch: libglusterfs / src / glusterfs.h:117</div><div><br></div>#define GLUSTERFS_INTERNAL_FOP_KEY "glusterfs-internal-fop"<div>
<br></div><div>It should be:</div><div><br></div><div>#define GLUSTERFS_INTERNAL_FOP_KEY "glusterfs.internal-fop"<br></div><div><br></div><div>It's a typo I think.</div><div><br></div><div><br></div><div>Best regard,</div>
<div>Johnny Chang</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-03-10 15:03 GMT+08:00 張為超 <span dir="ltr"><<a href="mailto:j1899j1899@gmail.com" target="_blank">j1899j1899@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>Here's my command history and some error messages in the log.</div><div>I hope these are helpful.</div>
<div><br></div><div><div>*** .cmd_log_history: ***</div><div>[2014-03-10 06:05:04.176225] : volume create gv1 192.168.13.93:/brick 192.168.5.198:/brick : SUCCESS</div>
<div>[2014-03-10 06:05:07.048590] : volume start gv1 : SUCCESS</div><div>[2014-03-10 06:08:20.781311] : volume add-brick gv1 192.168.14.36:/brick : SUCCESS</div><div>[2014-03-10 06:09:10.373150] : volume remove-brick gv1 192.168.13.93:/brick start : SUCCESS</div>
<div>[2014-03-10 06:11:24.400824] : volume remove-brick gv1 192.168.13.93:/brick status : SUCCESS<br></div><div>[2014-03-10 06:12:30.043228] : volume remove-brick gv1 192.168.13.93:/brick commit : SUCCESS</div><div>[2014-03-10 06:12:34.567630] : volume remove-brick gv1 192.168.5.198:/brick status : SUCCESS</div>
<div><br></div><div><br></div><div>*** volume info after last command ***</div><div>Volume Name: gv1</div><div>Type: Distribute</div><div>Volume ID: 0d889dae-1a28-4f07-8991-0bb7a8c9da33</div><div>Status: Started</div><div>
Number of Bricks: 2</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: 192.168.5.198:/brick</div><div>Brick2: 192.168.14.36:/brick</div><div><br></div><div><br></div><div>*** Because brick (192.168.5.198:/brick) are full, some files are write to the other brick (192.168.13.93:/brick) ***</div>
<div>*** I found that the missing files are the files "linked" from the full brick (192.168.5.198:/brick) ***</div><div>*** Here's some error messages about the missing files. Do anyone knows why setting xattrs fails? ***<br>
</div><div>*** <a href="http://192.168.13.93" target="_blank">192.168.13.93</a>: brick.log ***<br></div><div>[2014-03-10 06:07:53.017207] W [posix-helpers.c:737:posix_handle_pair] 0-gv1-posix: Extended attributes not supported (try remounting brick with 'user_xattr' flag)</div>
<div>[2014-03-10 06:07:53.029126] E [posix.c:1751:posix_create] 0-gv1-posix: setting xattrs on /brick/file-15 failed (Operation not supported)</div><div>[2014-03-10 06:08:04.218437] E [posix.c:1751:posix_create] 0-gv1-posix: setting xattrs on /brick/file-16 failed (Operation not supported)</div>
<div>[2014-03-10 06:12:27.801988] W [glusterfsd.c:1002:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x7f9e1d22a78d] (-->/lib64/libpthread.so.0(+0x7def) [0x7f9e1d8b1def] (-->/var/packages/GlusterfsMgmt/target/sbin/glusterfsd(glusterfs_sigwaiter+0xdc) [0x40816c]))) 0-: received signum (15), shutting down</div>
<div><br></div><div><br></div><div>*** <a href="http://192.168.13.93" target="_blank">192.168.13.93</a>: glusterd.vol.log ***</div><div>[2014-03-10 06:05:04.866511] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /brick on port 49152</div>
<div>[2014-03-10 06:05:04.868654] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600</div><div>[2014-03-10 06:05:04.868806] I [socket.c:3480:socket_init] 0-management: SSL support is NOT enabled</div>
<div>[2014-03-10 06:05:04.868833] I [socket.c:3495:socket_init] 0-management: using system polling thread</div><div>[2014-03-10 06:05:05.718552] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600</div>
<div>[2014-03-10 06:05:05.718819] I [socket.c:3480:socket_init] 0-management: SSL support is NOT enabled</div><div>[2014-03-10 06:05:05.718870] I [socket.c:3495:socket_init] 0-management: using system polling thread</div>
<div>[2014-03-10 06:05:05.719358] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div><div>[2014-03-10 06:05:05.719457] I [socket.c:2236:socket_event_handler] 0-transport: disconnecting now</div>
<div>[2014-03-10 06:08:17.350770] I [glusterd-brick-ops.c:370:__glusterd_handle_add_brick] 0-management: Received add brick req</div><div>[2014-03-10 06:08:17.381885] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div>
<div>[2014-03-10 06:08:17.381952] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div><div>[2014-03-10 06:08:17.381984] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div>
<div>[2014-03-10 06:08:17.383519] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div><div>[2014-03-10 06:08:17.383582] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div>
<div>[2014-03-10 06:08:17.383613] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div><div>[2014-03-10 06:08:17.384776] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div>
<div>[2014-03-10 06:09:02.703267] I [glusterd-brick-ops.c:593:__glusterd_handle_remove_brick] 0-management: Received rem brick req</div><div>[2014-03-10 06:09:02.704150] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div>
<div>[2014-03-10 06:09:02.705531] I [glusterd-utils.c:7658:glusterd_generate_and_set_task_id] 0-management: Generated task-id b3b5bf8f-86e6-4ded-8c5a-a951db48d705 for key remove</div><div>[2014-03-10 06:09:02.707226] I [glusterd-op-sm.c:4168:glusterd_bricks_select_remove_brick] 0-management: force flag is not set</div>
<div>[2014-03-10 06:09:02.707904] W [dict.c:480:dict_unref] (-->/var/packages/GlusterfsMgmt/target/lib/glusterfs/3.4git/xlator/mgmt/glusterd.so(gd_commit_op_phase+0xe1) [0x7f589</div><div>[2014-03-10 06:09:02.708634] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div>
<div>[2014-03-10 06:09:02.755359] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div><div>[2014-03-10 06:09:02.755436] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div>
<div>[2014-03-10 06:09:02.755483] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div><div>[2014-03-10 06:09:02.757110] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div>
<div>[2014-03-10 06:09:02.757218] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div><div>[2014-03-10 06:09:02.757252] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div>
<div>[2014-03-10 06:09:08.870637] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600</div><div>[2014-03-10 06:09:08.870857] I [socket.c:3480:socket_init] 0-management: SSL support is NOT enabled</div>
<div>[2014-03-10 06:09:08.870908] I [socket.c:3495:socket_init] 0-management: using system polling thread</div><div>[2014-03-10 06:11:15.089162] I [glusterd-handshake.c:358:__server_event_notify] 0-: received defrag status updated</div>
<div>[2014-03-10 06:11:15.094016] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available)</div><div>[2014-03-10 06:11:15.094104] W [socket.c:1962:__socket_proto_state_machine] 0-management: reading from socket failed. Error (No data available), peer (/tmp/glusterfs-rebala</div>
<div>[2014-03-10 06:11:15.204709] I [mem-pool.c:541:mem_pool_destroy] 0-management: size=2236 max=0 total=0</div><div>[2014-03-10 06:11:15.204829] I [mem-pool.c:541:mem_pool_destroy] 0-management: size=124 max=0 total=0</div>
<div>[2014-03-10 06:12:27.789846] I [glusterd-brick-ops.c:593:__glusterd_handle_remove_brick] 0-management: Received rem brick req</div><div>[2014-03-10 06:12:27.790629] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div>
<div>[2014-03-10 06:12:27.793175] I [glusterd-op-sm.c:4168:glusterd_bricks_select_remove_brick] 0-management: force flag is not set</div><div>[2014-03-10 06:12:27.794029] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div>
<div>[2014-03-10 06:12:27.794700] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available)</div><div>[2014-03-10 06:12:27.801447] E [glusterd-utils.c:1618:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/1c5239eb7451335304f0aee8712e7cdf.socket err</div>
<div>[2014-03-10 06:12:27.825757] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div><div>[2014-03-10 06:12:27.825819] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div>
<div>[2014-03-10 06:12:27.826741] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div><div>[2014-03-10 06:12:27.826804] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick</div>
<div>[2014-03-10 06:12:28.710599] I [mem-pool.c:541:mem_pool_destroy] 0-management: size=2236 max=0 total=0</div><div>[2014-03-10 06:12:28.710705] I [mem-pool.c:541:mem_pool_destroy] 0-management: size=124 max=0 total=0</div>
<div>[2014-03-10 06:12:28.710894] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /brick on port 49152</div><div>[2014-03-10 06:12:28.713848] W [socket.c:514:__socket_rwv] 0-socket.management: writev failed (Broken pipe)</div>
<div>[2014-03-10 06:12:28.713985] I [socket.c:2236:socket_event_handler] 0-transport: disconnecting now</div></div><div><br></div><div><br></div><div>Thanks.</div><div>Johnny Chang</div></div><div class="gmail_extra"><br>
<br><div class="gmail_quote">2014-03-04 17:31 GMT+08:00 張為超 <span dir="ltr"><<a href="mailto:j1899j1899@gmail.com" target="_blank">j1899j1899@gmail.com</a>></span>:<div><div class="h5"><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi all,<div><br></div><div>I have 3 peers (peer-A, peer-B and peer-C). I tried to use add-brick and remove-brick to replace peers.</div><div>(version: glusterfs 3.4)</div><div><br></div><div>What I did:</div>
<div><ol><li>created a distribute volume with two 10-GB bricks (peer-A:/brick and peer-B:/brick. Actually they are 9.7 GB after ext4 formatting).</li><li>mount it and write 16 1-GB files in to it (command: seq 16 | xargs -i dd if=/dev/zero of=/mnt/file-{} bs=1G count=1).</li>
<li>add peer-C:/brick (also 10-GB) to this volume.</li><li>execute remove peer-A:/brick start.</li><li>check remove status and wait until all of the hosts are completed.</li><li>execute remove peer-A:/brick commit.</li></ol>
After step 6, I lost 2 files in the volume.</div><div><br></div><div><br></div><div>I list the files in bricks after step 2 and step 5:</div><div>After step 2:<br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div>peer-A:/brick:</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-1</div></div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-12</div>
</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-14</div></div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-15</div></div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:08 file-16</div>
</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-3</div></div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-6</div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div><br></div><div>peer-B:/brick:</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-10</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-11</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-13</div>
<div>---------T 2 root root 0 Mar 4 17:07 file-15</div><div>---------T 2 root root 0 Mar 4 17:07 file-16</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-2</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-4</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-5</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-7</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-8</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-9</div></div><div><br></div></blockquote><div>After step 5:<br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div>peer-A:/brick:</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-15</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:08 file-16</div></div><div><br></div><div>peer-B:/brick:</div>
<div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-10</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-11</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-13</div>
<div>---------T 2 root root 1073741824 Mar 4 17:17 file-15</div><div>---------T 2 root root 1073741824 Mar 4 17:17 file-16</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-2</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-4</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-5</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-7</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-8</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-9</div></div><div><br></div><div>peer-C:/brick:</div><div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-1</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-12</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:07 file-14</div><div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:05 file-3</div>
<div>-rw-r--r-- 2 root root 1073741824 Mar 4 17:06 file-6</div></div></blockquote><div><br></div><div>After step 6, I lost file-15 and file-16 in the volume. </div><div>Anyone know why file-15 and file-16 are not moved to peer-C?</div>
<div>If it's caused by peer-B is full, why does the status show "completed"?</div><div><br></div><div><div>Node Rebalanced-files size scanned failures skipped status run-time in secs</div>
<div>--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</div><div>localhost 5 5.0GB 21 0 completed 126.00</div>
<div>localhost 5 5.0GB 21 0 completed 126.00</div><div>localhost 5 5.0GB 21 0 completed 126.00</div>
<div>localhost 5 5.0GB 21 0 completed 126.00</div></div><div><br></div><div><br></div><div>--</div><div>Best regards,</div><div>Johnny</div><div><a href="mailto:j1899j1899@gmail.com" target="_blank">j1899j1899@gmail.com</a></div>
<div><br></div></div>
</blockquote></div></div></div><br></div>
</blockquote></div><br></div>