<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font face="DejaVu Sans Mono">Just before updrading to 3.3final, we
had an rsync collision on our gluster filesystem which left us
with undeletable dirs. <br>
<br>
The transport is IPoIB over 4 bricks as shown below.<br>
<br>
$ gluster volume info<br>
<br>
Volume Name: gli<br>
Type: Distribute<br>
Volume ID: 76cc5e88-0ac4-42ac-a4a3-31bf2ba611d4<br>
Status: Started<br>
Number of Bricks: 4<br>
Transport-type: tcp,rdma<br>
Bricks:<br>
Brick1: pbs1ib:/bducgl<br>
Brick2: pbs2ib:/bducgl<br>
Brick3: pbs3ib:/bducgl<br>
Brick4: pbs4ib:/bducgl<br>
Options Reconfigured:<br>
performance.io-cache: on<br>
performance.quick-read: on<br>
performance.io-thread-count: 64<br>
<br>
The last 100 line of the gluster log is here:
<a class="moz-txt-link-rfc2396E" href="http://pastie.org/4036243"><http://pastie.org/4036243></a><br>
<br>
The problem started when I was running an rsync with --delete
while moving a large /home to the gluster dir to prune the target
of older cruft in a /home move.. A user directed a job to start
reading from a directory that was simultaneously being pruned and
all hell broke loose.<br>
<br>
Now we're left with a number of dirs and files that can't be
deleted. 'du' says there are files in an apparently empty dir:<br>
<br>
root@bduc-login:/gl/tvanerp/adni<br>
561 $ du -sh *<br>
40K dicom<br>
<br>
altho 'ls' can't see them from the client;<br>
<br>
root@bduc-login:/gl/tvanerp/adni<br>
562 $ ls -lat dicom<br>
total 40<br>
drwxrwxr-x 3 tvanerp tvanerp 94214 Jun 5 23:40 ./<br>
drwxrwxr-x 3 tvanerp tvanerp 72 Apr 23 13:11 ../<br>
<br>
<br>
From the bricks you can see that there are still files there:<br>
Tue Jun 05 23:18:02 [3.06 2.42 1.65][457.48/606]
root@bduc-login:~<br>
556 $ ssh pbs2 'ls -lat /bducgl/tvanerp/adni/dicom'<br>
total 24<br>
drwxrwxr-x 3 7335 7335 28672 2012-06-05 22:59 .<br>
drwxr-xr-x 2 7335 7335 8192 2012-06-05 14:50
128_S_0947_20080422_MPRAGE<br>
drwxrwxr-x 3 7335 7335 18 2012-04-23 13:11 ..<br>
<br>
Tue Jun 05 23:18:26 [2.82 2.42 1.67][457.62/606]
root@bduc-login:~<br>
557 $ ssh pbs3 'ls -lat /bducgl/tvanerp/adni/dicom'<br>
total 28<br>
drwxrwxr-x 3 7335 7335 28672 2012-06-05 23:00 .<br>
drwxr-xr-x 2 7335 7335 12288 2012-06-05 14:52
128_S_0947_20080422_MPRAGE<br>
drwxrwxr-x 3 7335 7335 18 2012-04-23 13:11 ..<br>
<br>
Tue Jun 05 23:18:51 [2.73 2.43 1.69][457.7/606] root@bduc-login:~<br>
558 $ ssh pbs4 'ls -lat /bducgl/tvanerp/adni/dicom'<br>
total 32<br>
drwxrwxr-x 3 7335 7335 36864 2012-06-05 22:59 .<br>
drwxr-xr-x 2 7335 7335 12288 2012-06-05 14:50
128_S_0947_20080422_MPRAGE<br>
---------T 2 root root 0 2012-05-20 18:35
adni_pib_subjects.txt<br>
drwxrwxr-x 3 7335 7335 18 2012-04-23 13:11 ..<br>
<br>
but the client is not able to see/delete them, even as root.<br>
<br>
Suggestions?<br>
<br>
harry<br>
<br>
<br>
<br>
<br>
</font>
<pre class="moz-signature" cols="72">--
Harry Mangalam,Research Computing, OIT,
Rm 225 MSTB, UC Irvine [mailcode 2225]
Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
Lat/Long: (33.642025,-117.844414) [paste into Google Maps]
--</pre>
</body>
</html>