<div dir="ltr"><div>I profiled node22, I found that most latency comes from setxattr, where node23 & node22 comes from lookup and locks. any one could help?<br><br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br>
--------- ----------- ----------- ----------- ------------ ----<br> 0.00 0.00 us 0.00 us 0.00 us 2437540 FORGET<br> 0.00 0.00 us 0.00 us 0.00 us 252684 RELEASE<br>
0.00 0.00 us 0.00 us 0.00 us 2226292 RELEASEDIR<br> 0.00 38.00 us 37.00 us 40.00 us 4 FGETXATTR<br> 0.00 66.16 us 15.00 us 13139.00 us 596 GETXATTR<br>
0.00 239.14 us 58.00 us 126477.00 us 1967 LINK<br> 0.00 51.85 us 14.00 us 8298.00 us 19045 STAT<br> 0.00 165.50 us 9.00 us 212057.00 us 20544 READDIR<br>
0.00 1827.92 us 184.00 us 150298.00 us 2084 RENAME<br> 0.00 49.14 us 12.00 us 5908.00 us 189019 STATFS<br> 0.00 84.63 us 14.00 us 96016.00 us 163405 READ<br>
0.00 29968.76 us 156.00 us 1073902.00 us 3115 CREATE<br> 0.00 1340.25 us 6.00 us 7415357.00 us 248141 FLUSH<br> 0.00 1616.76 us 32.00 us 13865122.00 us 229190 FTRUNCATE<br>
0.01 1807.58 us 19.00 us 55480776.00 us 249569 OPEN<br> 0.01 1875.11 us 10.00 us 8842171.00 us 465197 FSTAT<br> 0.05 393296.28 us 52.00 us 56856581.00 us 9057 UNLINK<br>
0.07 32291.01 us 192.00 us 9638107.00 us 156081 RMDIR<br> 0.08 18339.18 us 140.00 us 5313885.00 us 337862 MKNOD<br> 0.09 2904.39 us 18.00 us 51724741.00 us 2226290 OPENDIR<br>
0.15 4708.15 us 27.00 us 55115760.00 us 2334864 SETXATTR<br> 0.18 8965.91 us 68.00 us 26465968.00 us 1513280 FXATTROP<br> 0.21 3465.29 us 74.00 us 58580783.00 us 4506602 XATTROP<br>
0.28 4801.16 us 44.00 us 49643138.00 us 4436847 READDIRP<br> 0.37 5935.92 us 7.00 us 56449083.00 us 4611760 ENTRYLK<br> 1.02 4226.58 us 33.00 us 63494729.00 us 18092335 WRITE<br>
1.50 2734.50 us 6.00 us 185109908.00 us 40971541 INODELK<br> 4.75 348602.30 us 5.00 us 2185602946.00 us 1019332 FINODELK<br><b> 14.98 33957.49 us 14.00 us 59261447.00 us 32998211 LOOKUP<br>
26.30 807063.74 us 150.00 us 68086266.00 us 2438422 MKDIR<br> 49.95 457402.30 us 20.00 us 67894186.00 us 8171751 SETATTR</b><br><br> Duration: 353678 seconds<br> Data Read: 21110920120 bytes<br>
Data Written: 2338403381483 bytes<br><br>here is node23 <br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br> --------- ----------- ----------- ----------- ------------ ----<br>
0.00 0.00 us 0.00 us 0.00 us 22125898 FORGET<br> 0.00 0.00 us 0.00 us 0.00 us 89286732 RELEASE<br> 0.00 0.00 us 0.00 us 0.00 us 32865496 RELEASEDIR<br>
0.00 35.50 us 23.00 us 48.00 us 2 FGETXATTR<br> 0.00 164.04 us 29.00 us 749181.00 us 39320 FTRUNCATE<br> 0.00 483.71 us 8.00 us 2688755.00 us 39288 LK<br>
0.00 419.61 us 48.00 us 2183971.00 us 274939 LINK<br> 0.00 970.55 us 145.00 us 2471745.00 us 293435 RENAME<br> 0.00 1346.63 us 35.00 us 4462970.00 us 243238 SETATTR<br>
0.01 285.51 us 25.00 us 2588685.00 us 3459436 SETXATTR<br> 0.03 323.11 us 5.00 us 2074581.00 us 6977304 READDIR<br> 0.05 12200.60 us 84.00 us 3943421.00 us 287979 RMDIR<br>
0.07 592.75 us 7.00 us 3592073.00 us 8129847 STAT<br> 0.07 6938.50 us 49.00 us 3268036.00 us 705818 UNLINK<br> 0.08 19468.78 us 149.00 us 3664022.00 us 276310 MKNOD<br>
0.09 763.31 us 8.00 us 3396903.00 us 8731725 STATFS<br> 0.09 1715.79 us 4.00 us 5626912.00 us 3902746 FLUSH<br> 0.10 4614.74 us 9.00 us 5835691.00 us 1574923 FSTAT<br>
0.10 1189.55 us 13.00 us 6043163.00 us 6129885 OPENDIR<br> 0.10 19729.66 us 131.00 us 4112832.00 us 376286 CREATE<br> 0.13 328.26 us 24.00 us 2410049.00 us 29091424 WRITE<br>
0.20 2107.64 us 10.00 us 5765196.00 us 6675496 GETXATTR<br> 0.28 5317.38 us 14.00 us 7549301.00 us 3798543 OPEN<br> 0.71 7042.79 us 47.00 us 5848284.00 us 7125716 READDIRP<br>
0.80 743.88 us 10.00 us 7979373.00 us 76781383 READ<br> 0.93 1802.29 us 60.00 us 11040319.00 us 36501360 FXATTROP<br> 1.76 36083.12 us 141.00 us 3548175.00 us 3458135 MKDIR<br>
1.83 5046.35 us 70.00 us 8120221.00 us 25765615 XATTROP<br> 11.74 12896.99 us 4.00 us 2141920969.00 us 64590600 FINODELK<br> 15.43 11171.78 us 5.00 us 909115697.00 us 98040443 ENTRYLK<br>
25.46 12945.21 us 5.00 us 110968164.00 us 139545956 INODELK<br> 39.91 9656.48 us 10.00 us 8137517.00 us 293268060 LOOKUP<br><br></div>here is node24<br><br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br>
--------- ----------- ----------- ----------- ------------ ----<br> 0.00 0.00 us 0.00 us 0.00 us 22124594 FORGET<br> 0.00 0.00 us 0.00 us 0.00 us 89290582 RELEASE<br>
0.00 0.00 us 0.00 us 0.00 us 26657287 RELEASEDIR<br> 0.00 47.00 us 47.00 us 47.00 us 1 FGETXATTR<br> 0.00 308.67 us 8.00 us 1405672.00 us 39285 LK<br>
0.00 745.82 us 32.00 us 1690066.00 us 86586 FTRUNCATE<br> 0.00 388.58 us 49.00 us 1348668.00 us 274927 LINK<br> 0.00 1008.11 us 158.00 us 2443763.00 us 293423 RENAME<br>
0.01 1094.49 us 31.00 us 2857159.00 us 290615 SETATTR<br> 0.02 304.24 us 24.00 us 2878581.00 us 3506688 SETXATTR<br> 0.03 279.83 us 5.00 us 3716543.00 us 6977266 READDIR<br>
0.05 10919.43 us 83.00 us 5075633.00 us 287979 RMDIR<br> 0.05 692.45 us 12.00 us 3951452.00 us 4692109 OPENDIR<br> 0.06 465.87 us 6.00 us 3726826.00 us 8238785 STAT<br>
0.07 1187.15 us 14.00 us 5361516.00 us 3626802 GETXATTR<br> 0.07 6308.14 us 50.00 us 4281153.00 us 705476 UNLINK<br> 0.07 16729.47 us 148.00 us 3238674.00 us 276299 MKNOD<br>
0.08 553.69 us 8.00 us 2721668.00 us 8744855 STATFS<br> 0.09 1462.59 us 4.00 us 5488045.00 us 3903587 FLUSH<br> 0.10 16979.85 us 130.00 us 3471136.00 us 376279 CREATE<br>
0.12 4818.36 us 9.00 us 6101767.00 us 1577172 FSTAT<br> 0.15 315.32 us 24.00 us 3801518.00 us 29090837 WRITE<br> 0.19 2539.98 us 48.00 us 4657386.00 us 4586952 READDIRP<br>
0.23 3794.04 us 15.00 us 6487700.00 us 3798788 OPEN<br> 0.37 393.76 us 10.00 us 3284611.00 us 58491958 READ<br> 0.88 1524.40 us 60.00 us 7456834.00 us 36097324 FXATTROP<br>
1.63 4429.64 us 72.00 us 7194041.00 us 22984938 XATTROP<br> 1.74 31485.11 us 143.00 us 4705647.00 us 3458000 MKDIR<br> 2.08 2010.98 us 4.00 us 7669056.00 us 64626004 FINODELK<br>
18.35 11708.39 us 4.00 us 7193745.00 us 98037767 ENTRYLK<br> 31.62 14170.24 us 5.00 us 7194060.00 us 139544869 INODELK<br> 41.94 9273.78 us 10.00 us 7193886.00 us 282853490 LOOKUP<br>
<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 22, 2014 at 12:05 PM, Mingfan Lu <span dir="ltr"><<a href="mailto:mingfan.lu@gmail.com" target="_blank">mingfan.lu@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>I have a volume (distribute-replica (*3)), today i found an interesting problem<br><br></div>
<div>node22 node23 and node24 are the replica-7 from client A<br></div><div>but the annoying thing is when I create dir or write file from client to replica-7,<br>
</div><div><br> date;dd if=/dev/zero of=49 bs=1MB count=120<br>Wed Jan 22 11:51:41 CST 2014<br>120+0 records in<br>120+0 records out<br>120000000 bytes (120 MB) copied, 1.96257 s, 61.1 MB/s<br></div><div><br></div><div>but I could only find node23 & node24 have the find<br>
</div><div>---------------<br>node23,node24<br>---------------<br>/mnt/xfsd/test-volume/test/49<br><br></div><div>in clientA, I use find command<br></div><div><br></div><div>I use another machine as client B, and mount the test volume (newly mounted)<br>
</div><div>to run<b> find /mnt/xfsd/test-volume/test/49</b><br><br></div><div>from Client A, the three nodes have the file now.<br><br>---------------<br>node22,node23.node24<br>---------------<br>/mnt/xfsd/test-volume/test/49<br>
<br></div><div>but in Client A, I delete the file /mnt/xfsd/test-volume/test/49, node22 still have the file in brick.<br><br>---------------<br>node22<br>---------------<br>/mnt/xfsd/test-volume/test/49<br></div><div><br>
</div><div>but if i delete the new created files from Client B )<br></div><div>my question is why node22 have no newly created/write dirs/files? I have to use find to trigger the self-heal to fix that?<br></div><br></div>
from ClientA's log, I find something like:<br><br> I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-test-volume-replicate-7: no active sinks for performing self-heal on file /test/49<br><br></div>It is harmless for it is information level?<br>
<div><br><div><div>I also see something like:<br>[2014-01-19 10:23:48.422757] E [afr-self-heal-entry.c:2376:afr_sh_post_nonblocking_entry_cbk] 0-test-volume-replicate-7: Non Blocking entrylks failed for /test/video/2014/01.<br>
[2014-01-19 10:23:48.423042] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-test-volume-replicate-7: background entry self-heal failed on /test/video/2014/01<br><br></div><div><br><br></div><div><br></div>
</div></div></div>
</blockquote></div><br></div>