<p> I use glusterfs in a cluster system (configured as: dht->afr->client->server->iothreads->locks->posix), after days running, it is stable, but with a poor porformance (slower thann NFS exported from only one server), and most important is that a bug came to me these days. This is really an emergency, so I need your help!</p>
<div>What is the BUG? In this system, I use mvapich+blcr for task checkpoint and restore. I don't know how mvapich works, but I am sure it used glusterfs in my case. When using glusterfs in checkpointing a task, it created one ckpt file for each proccess of the task, all the ckpt files placed in directory called 1, and it will create a symbol link called 0 pointing to directory 1. There is example, fortest is username, .ckpt is the ckpt file directory for this user, 1972 is the task id, 0 is the symbol link and bt.C.64-19.ckpt is a ckpt file the task's 19th proccess</div>
<div>[fortest@gfsclient02 1972]$ pwd<br>/mnt/glusterfs/.ckpt/1972<br>[fortest@gfsclient02 1972]$ ll<br>total 132<br>lrwxrwxrwx 1 fortest fortest 31 Sep 4 17:09 0 -> /mnt/glusterfs/fortest/.ckpt/1972/1<br>drwx------ 2 fortest fortest 65536 Sep 4 20:06 1</div>
<div>[fortest@gfsclient02 1972]$ ls 1/<br>bt.C.64-0.ckpt bt.C.64-21.ckpt bt.C.64-33.ckpt bt.C.64-45.ckpt bt.C.64-57.ckpt<br>bt.C.64-10.ckpt bt.C.64-22.ckpt bt.C.64-34.ckpt bt.C.64-46.ckpt bt.C.64-58.ckpt<br>bt.C.64-11.ckpt bt.C.64-23.ckpt bt.C.64-35.ckpt bt.C.64-47.ckpt bt.C.64-59.ckpt<br>
bt.C.64-12.ckpt bt.C.64-24.ckpt bt.C.64-36.ckpt bt.C.64-48.ckpt bt.C.64-5.ckpt<br>bt.C.64-13.ckpt bt.C.64-25.ckpt bt.C.64-37.ckpt bt.C.64-49.ckpt bt.C.64-60.ckpt<br>bt.C.64-14.ckpt bt.C.64-26.ckpt bt.C.64-38.ckpt bt.C.64-4.ckpt bt.C.64-61.ckpt<br>
bt.C.64-15.ckpt bt.C.64-27.ckpt bt.C.64-39.ckpt bt.C.64-50.ckpt bt.C.64-62.ckpt<br>bt.C.64-16.ckpt bt.C.64-28.ckpt bt.C.64-3.ckpt bt.C.64-51.ckpt bt.C.64-63.ckpt<br>bt.C.64-17.ckpt bt.C.64-29.ckpt bt.C.64-40.ckpt bt.C.64-52.ckpt bt.C.64-6.ckpt<br>
bt.C.64-18.ckpt bt.C.64-2.ckpt bt.C.64-41.ckpt bt.C.64-53.ckpt bt.C.64-7.ckpt<br>bt.C.64-19.ckpt bt.C.64-30.ckpt bt.C.64-42.ckpt bt.C.64-54.ckpt bt.C.64-8.ckpt<br>bt.C.64-1.ckpt bt.C.64-31.ckpt bt.C.64-43.ckpt bt.C.64-55.ckpt bt.C.64-9.ckpt<br>
bt.C.64-20.ckpt bt.C.64-32.ckpt bt.C.64-44.ckpt bt.C.64-56.ckpt</div>
<div> </div>
<div>When the task need to be restored, mvapich will read the ckpt file from 0 (the symbol link) and restore the task! All this perform smoothly in NFS, but in glusterfs it will output following messages. However sometimes task restoring can finish at last, while others can't almost with the same messages. I have verifed the missing files mvapich outputed was indeed there. Another useful tips is that fewer gluster client doing the task, few times it would be came to this bug when task restoring. And startup glusterfs without direct-io could not help too. </div>
<p>OUTPUT OF THE TASK WHEN RESTORE:</p>
<p>19: Restart: path /mnt/glusterfs/fortest/.ckpt/1972/0/bt.C.64-19.ckpt: No such file or directory20: Restart: path /mnt/glusterfs/fortest/.ckpt/1972/0/bt.C.64-20.ckpt: No such file or directorysrun: error: gfsclient10: task[19-20]: Exited with exit code 1<br>
21: Restart: path /mnt/glusterfs/fortest/.ckpt/1972/0/bt.C.64-21.ckpt: No such file or directory18: Restart: path /mnt/glusterfs/fortest/.ckpt/1972/0/bt.C.64-18.ckpt: No such file or directorysrun: error: gfsclient10: task21: Exited with exit code 1<br>
srun: error: cn010: task18: Exited with exit code 1<br>17: Restart: path /mnt/glusterfs/fortest/.ckpt/1972/0/bt.C.64-17.ckpt: No such file or directorysrun: error: gfsclient10: task17: Exited with exit code 1<br>23: Restart: path /mnt/glusterfs/fortest/.ckpt/1972/0/bt.C.64-23.ckpt: No such file or directory22: Restart: path /mnt/glusterfs/fortest/.ckpt/1972/0/bt.C.64-22.ckpt: No such file or directorysrun: error: gfsclient10: task23: Exited with exit code 1<br>
srun: error: cn010: task[16,22]: Exited with exit code 1<br>16: Restart: path /mnt/glusterfs/fortest/.ckpt/1972/0/bt.C.64-16.ckpt: No such file or directory</p>
<p><br>I use "debug/trace" and start the gluster with "-L DEBUG", and got the following logs when the ckpt can't to be found:</p>
<div>[2009-09-04 17:12:35] N [trace.c:1290:trace_readlink] tr0: 174536: (loc {path=/fortest/.ckp<br>t/1972/0, ino=1380450540}, size=4096)<br>[2009-09-04 17:12:35] N [trace.c:484:trace_readlink_cbk] tr0: 174536: (op_ret=31, op_errno=<br>
0, buf=/mnt/glusterfs/fortest/.ckpt/1972/1)<br>[2009-09-04 17:12:35] E [fuse-bridge.c:987:fuse_readlink_cbk] glusterfs-fuse: 174536: /fortest/<br>.ckpt/1972/0 => /mnt/glusterfs/fortest/.ckpt/1972/1 @ 1252055555<br>[2009-09-04 17:12:35] N [trace.c:1245:trace_lookup] tr0: 174537: (loc {path=/fortest/.ckpt/<br>
1972/1, ino=0})<br>[2009-09-04 17:12:35] N [trace.c:513:trace_lookup_cbk] tr0: 174508: (op_ret=0, ino=0, *buf<br>{st_dev=2065, st_ino=7068450884, st_mode=40700, st_nlink=2, st_uid=1001, st_gid=1001, st_rd<br>ev=0, st_size=65536, st_blksize=4096, st_blocks=256})<br>
[2009-09-04 17:12:35] E [fuse-bridge.c:255:fuse_loc_fill] glusterfs-fuse: inode_path failed for<br> 8003256399/bt.C.64-22.ckpt @ 1252055555<br>[2009-09-04 17:12:35] W [fuse-bridge.c:436:fuse_lookup] glusterfs-fuse: 174539: LOOKUP 80032563<br>
99/bt.C.64-22.ckpt (fuse_loc_fill() failed)<br>[2009-09-04 17:12:35] N [trace.c:513:trace_lookup_cbk] tr0: 174522: (op_ret=0, ino=0, *buf<br>{st_dev=2065, st_ino=7068450884, st_mode=40700, st_nlink=2, st_uid=1001, st_gid=1001, st_rd<br>
ev=0, st_size=65536, st_blksize=4096, st_blocks=256})<br>[2009-09-04 17:12:35] E [fuse-bridge.c:255:fuse_loc_fill] glusterfs-fuse: inode_path failed for<br> 8003256399/bt.C.64-16.ckpt @ 1252055555</div>
<div> </div>
<div> </div>
<div> </div>