<div dir="ltr">Hi, all.<div><br></div><div> I thought I had everything set correctly on my volume, but something is wrong. Here is the volume, made of 4 bricks:</div><div><br></div><div><div><font face="courier new, monospace">Volume Name: gf2</font></div>
<div><font face="courier new, monospace">Type: Distributed-Replicate</font></div><div><font face="courier new, monospace">Volume ID: a9e64630-9166-4957-8243-e2933791b24b</font></div><div><font face="courier new, monospace">Status: Started</font></div>
<div><font face="courier new, monospace">Number of Bricks: 2 x 2 = 4</font></div><div><font face="courier new, monospace">Transport-type: tcp</font></div><div><font face="courier new, monospace">Bricks:</font></div><div><font face="courier new, monospace">Brick1: gf2ibp-1:/mnt/d0-0</font></div>
<div><font face="courier new, monospace">Brick2: gf2ibp-1r:/mnt/d0-0</font></div><div><font face="courier new, monospace">Brick3: gf2ibp-2:/mnt/d0-0</font></div><div><font face="courier new, monospace">Brick4: gf2ibp-2r:/mnt/d0-0</font></div>
<div><br></div><div style>I have Volume gf2 mounted by a computer we call "rcapps"</div><div style>About 6 TB have been written to the volume.</div><div style>When I look at /mnt/d0-0 on all 4 bricks, 3 look correct, but</div>
<div style> Brick1 only has 48GB written to it.</div><div style> Brick2, which should replicate Brick1, has 4TB.</div><div style> Brick3 and Brick4 seem to have the same amount of data.</div><div style><br></div>
<div style>The status of the volume looks correct:</div><div style><br></div><div style><div><font face="courier new, monospace">gluster> volume status gf2</font></div><div><font face="courier new, monospace">Status of volume: gf2</font></div>
<div><font face="courier new, monospace">Gluster process<span class="" style="white-space:pre"> </span>Port<span class="" style="white-space:pre"> </span>Online<span class="" style="white-space:pre"> </span>Pid</font></div>
<div><font face="courier new, monospace">------------------------------------------------------------------------------</font></div><div><font face="courier new, monospace">Brick gf2ibp-1:/mnt/d0-0<span class="" style="white-space:pre"> </span>24011<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>30754</font></div>
<div><font face="courier new, monospace">Brick gf2ibp-1r:/mnt/d0-0<span class="" style="white-space:pre"> </span>24011<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>17824</font></div>
<div><font face="courier new, monospace">Brick gf2ibp-2:/mnt/d0-0<span class="" style="white-space:pre"> </span>24011<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>31516</font></div>
<div><font face="courier new, monospace">Brick gf2ibp-2r:/mnt/d0-0<span class="" style="white-space:pre"> </span>24011<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>29119</font></div>
<div><font face="courier new, monospace">NFS Server on localhost<span class="" style="white-space:pre"> </span>38467<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>30760</font></div>
<div><font face="courier new, monospace">Self-heal Daemon on localhost<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>30766</font></div>
<div><font face="courier new, monospace">NFS Server on gf2ibp-2<span class="" style="white-space:pre"> </span>38467<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>31522</font></div>
<div><font face="courier new, monospace">Self-heal Daemon on gf2ibp-2<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>31528</font></div>
<div><font face="courier new, monospace">NFS Server on gf2ibp-2r<span class="" style="white-space:pre"> </span>38467<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>29125</font></div>
<div><font face="courier new, monospace">Self-heal Daemon on gf2ibp-2r<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>29131</font></div>
<div><font face="courier new, monospace">NFS Server on gf2ibp-1r<span class="" style="white-space:pre"> </span>38467<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>17830</font></div>
<div><font face="courier new, monospace">Self-heal Daemon on gf2ibp-1r<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>17836</font></div>
</div><div style><font face="courier new, monospace"><br></font></div><div style>I then saw I had the firewall turned on for brick2 (even though it could be written) which I then turned off.</div><div style>I thought I should try to heal the volume but when I tried this through the gluster console, the operation failed.</div>
<div style>In the log file I see (it can't get a lock which is held by itself?):</div><div style><br></div><div style><div><font face="courier new, monospace">[2013-02-21 09:24:39.501612] I [glusterd-volume-ops.c:492:glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gf2</font></div>
<div><font face="courier new, monospace">[2013-02-21 09:24:39.501732] E [glusterd-utils.c:277:glusterd_lock] 0-glusterd: Unable to get lock for uuid: f5edea20-9467-48ed-b4f1-dc566a9b6d02, lock held by: f5edea20-9467-48ed-b4f1-dc566a9b6d02</font></div>
<div><font face="courier new, monospace">[2013-02-21 09:24:39.501759] E [glusterd-handler.c:458:glusterd_op_txn_begin] 0-management: Unable to acquire local lock, ret: -1</font></div><div><font face="courier new, monospace"><br>
</font></div><div style><font face="arial, helvetica, sans-serif">And here is what I see in cli.log, which I can't interpret.</font></div><div style><font face="arial, helvetica, sans-serif"><br></font></div><div style>
<div><font face="courier new, monospace">2013-02-21 09:31:38.689316] W [cli-rl.c:116:cli_rl_process_line] 0-glusterfs: failed to process line</font></div><div><font face="courier new, monospace">[2013-02-21 09:31:48.952950] I [cli-rpc-ops.c:5928:gf_cli3_1_heal_volume_cbk] 0-cli: Received resp to heal volume</font></div>
<div><font face="courier new, monospace">[2013-02-21 09:31:48.953366] W [dict.c:2339:dict_unserialize] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x120) [0x333440f8b0] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x333440f0b5] (-->gluster(gf_cli3_1_heal_volume_cbk+0x2e3) [0x41ca43]))) 0-dict: buf is null!</font></div>
<div><font face="courier new, monospace">[2013-02-21 09:31:48.953410] E [cli-rpc-ops.c:5968:gf_cli3_1_heal_volume_cbk] 0-: Unable to allocate memory</font></div><div><font face="courier new, monospace">[2013-02-21 09:31:48.953490] W [cli-rl.c:116:cli_rl_process_line] 0-glusterfs: failed to process line</font></div>
<div><font face="courier new, monospace">[2013-02-21 09:31:56.419708] I [cli-rpc-ops.c:5928:gf_cli3_1_heal_volume_cbk] 0-cli: Received resp to heal volume</font></div><div><font face="courier new, monospace">[2013-02-21 09:31:56.419859] W [dict.c:2339:dict_unserialize] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x120) [0x333440f8b0] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x333440f0b5] (-->gluster(gf_cli3_1_heal_volume_cbk+0x2e3) [0x41ca43]))) 0-dict: buf is null!</font></div>
<div><font face="courier new, monospace">[2013-02-21 09:31:56.419894] E [cli-rpc-ops.c:5968:gf_cli3_1_heal_volume_cbk] 0-: Unable to allocate memory</font></div><div><font face="courier new, monospace">[2013-02-21 09:31:56.419979] W [cli-rl.c:116:cli_rl_process_line] 0-glusterfs: failed to process line</font></div>
</div></div><div style><br></div><div style>Any ideas of what I should do next?</div><div style>Right now I have a pair of bricks that replicate fine and a pair that does not, in a distributed/replicated cluster.</div><div style>
I need to get get brick2 to send its files back to brick1.</div><div style><br></div><div style>Thanks in advance.</div><div style><br></div><div style>Matt Temple</div><div><br></div><div>------<br>Matt Temple<br>Director, Research Computing<br>
Dana-Farber Cancer Institute.<br></div>
</div></div>