<div dir="ltr">Hello,<div><br></div><div>I'm currently running a data migration, transferring about 450,000 directories to my GlusterFS cluster. Right now, I have one distributed-replicated volume with six bricks. </div>
<div><br></div><div>To migrate my files, I am running a simple shell script that does something like this</div><div><br></div><div>mkdir <new directory></div><div>rsync -a --inplace <old directory> <new directory><br clear="all">
<div>mv <old directory> <old_directory>.old</div><div>ln -s <new directory> <old directory></div><div><br></div><div>The old directory structure is on a non-GlusterFS NFS mount. It is too flat (all those directories are in one subdir) and the new structure is hashed with a maximum of 256 subdirs per directory.</div>
<div><br></div><div>To speed the migration along, I'm running it in parallel (with GNU Parallel) at a level of 16 (so I move 16 people simultaneously). As I am writing a lot of files to GlusterFS, I am using the native (fuse) client.</div>
<div><br></div><div>After a few minutes, I get a funky issue-- one of my file operations becomes totally unresponsive. Looking at the GlusterFS log on the client shows me messages like this:</div><div><br></div><div><div>
[2013-10-17 16:30:32.334799] W [dht-common.c:416:dht_lookup_dir_cbk] 0-UDS8-dht: /cache/ts/5a/5f/05/elchicote: gfid different on UDS8-replicate-0</div><div>[2013-10-17 16:30:32.335463] W [dht-common.c:416:dht_lookup_dir_cbk] 0-UDS8-dht: /cache/ts/5a/5f/05/elchicote: gfid different on UDS8-replicate-1</div>
</div><div><br></div><div>The hung call in this case is a mkdir of /cache/ts/5a/5f/05/elchicote.</div><div><br></div><div>Attempting to list /cache/ts/5f/05/elchicote on the client causes ls to hang.</div><div><br></div><div>
I haven't done any gfid munging here, so I am very concerned that GlusterFS 3.3.2 has this bug.</div><div><br></div>-- <br>Justin Dossey<br>CTO, PodOmatic<div><br></div>
</div></div>