<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=gb2312" http-equiv=Content-Type>
<STYLE>
BLOCKQUOTE {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px; MARGIN-LEFT: 2em
}
OL {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
UL {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
DIV.FoxDiv20130326074000788998 {
        LINE-HEIGHT: 1.5; FONT-FAMILY: & COLOR: #000000; FONT-SIZE: 10.5pt; 24494: ; amp: ; 36719: ; 38597: ; 40657:
}
BODY {
        LINE-HEIGHT: 1.5; FONT-FAMILY: ΢ÈíÑźÚ; COLOR: #000080; FONT-SIZE: 10.5pt
}
</STYLE>
<META name=GENERATOR content="MSHTML 8.00.7601.17514"></HEAD>
<BODY style="MARGIN: 10px" bgColor=#ffffff text=#000000>
<DIV>
<DIV>Hi Pranith:</DIV>
<DIV> </DIV>
<DIV> Thanks for your reply. I run Glusterfs 3.3¡£</DIV>
<DIV> I have a very important situation forgot to describe:</DIV>
<DIV> When the nfs server and client are both running CentOS 5.5,they work well. The problem appear only the nfs server run CentOS 5.5 and the
client run CentOS 6.3.</DIV>
<DIV> </DIV>
<DIV>The /var/log/messages: </DIV>
<DIV>Mar 12 18:04:41 localhost kernel: glusterfs invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: </DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Call Trace:</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: [<ffffffff800c723e>] out_of_memory+0x8e/0x2f3</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: [<ffffffff8002e22d>] __wake_up+0x38/0x4f</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: [<ffffffff8000f53f>] __alloc_pages+0x27f/0x308</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: [<ffffffff80008e9f>] __handle_mm_fault+0x73c/0x1039</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: [<ffffffff80066b55>] do_page_fault+0x4cb/0x874</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: [<ffffffff800f8935>] sys_epoll_wait+0x3b8/0x3f9</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: </DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Mem-info:</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Node 0 DMA per-cpu:</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 0 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 0 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 1 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 1 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 2 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 2 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 3 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 3 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 4 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 4 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 5 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 5 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 6 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 6 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 7 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 7 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Node 0 DMA32 per-cpu:</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 0 hot: high 186, batch 31 used:131</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 0 cold: high 62, batch 15 used:49</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 1 hot: high 186, batch 31 used:13</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 1 cold: high 62, batch 15 used:54</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 2 hot: high 186, batch 31 used:30</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 2 cold: high 62, batch 15 used:23</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 3 hot: high 186, batch 31 used:106</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 3 cold: high 62, batch 15 used:40</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 4 hot: high 186, batch 31 used:19</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 4 cold: high 62, batch 15 used:52</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 5 hot: high 186, batch 31 used:19</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 5 cold: high 62, batch 15 used:51</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 6 hot: high 186, batch 31 used:38</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 6 cold: high 62, batch 15 used:49</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 7 hot: high 186, batch 31 used:27</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 7 cold: high 62, batch 15 used:48</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Node 0 Normal per-cpu:</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 0 hot: high 186, batch 31 used:59</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 0 cold: high 62, batch 15 used:43</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 1 hot: high 186, batch 31 used:36</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 1 cold: high 62, batch 15 used:56</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 2 hot: high 186, batch 31 used:25</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 2 cold: high 62, batch 15 used:42</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 3 hot: high 186, batch 31 used:22</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 3 cold: high 62, batch 15 used:43</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 4 hot: high 186, batch 31 used:140</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 4 cold: high 62, batch 15 used:51</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 5 hot: high 186, batch 31 used:2</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 5 cold: high 62, batch 15 used:51</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 6 hot: high 186, batch 31 used:39</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 6 cold: high 62, batch 15 used:55</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 7 hot: high 186, batch 31 used:28</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: cpu 7 cold: high 62, batch 15 used:57</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Node 0 HighMem per-cpu: empty</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Free pages: 47116kB (0kB HighMem)</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Active:941872 inactive:355 dirty:0 writeback:0 unstable:0 free:11779 slab:40203 mapped-file:1 mapped-anon:938241 pagetables:3947</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Node 0 DMA free:10876kB min:80kB low:100kB high:120kB active:0kB inactive:0kB present:10476kB pages_scanned:0 all_unreclaimable? yes</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: lowmem_reserve[]: 0 2978 3988 3988</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Node 0 DMA32 free:28400kB min:24404kB low:30504kB high:36604kB active:2964256kB inactive:0kB present:3049956kB pages_scanned:15320246 all_unreclaimable? yes</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: lowmem_reserve[]: 0 0 1010 1010</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Node 0 Normal free:7840kB min:8276kB low:10344kB high:12412kB active:803516kB inactive:1292kB present:1034240kB pages_scanned:7795789 all_unreclaimable? yes</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: lowmem_reserve[]: 0 0 0 0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: lowmem_reserve[]: 0 0 0 0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Node 0 DMA: 3*4kB 0*8kB 5*16kB 3*32kB 5*64kB 3*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 10876kB</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Node 0 DMA32: 16*4kB 0*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 6*4096kB = 28400kB</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Node 0 Normal: 10*4kB 3*8kB 0*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 7840kB</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Node 0 HighMem: empty</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: 4366 pagecache pages</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Free swap = 0kB</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Total swap = 0kB</DIV>
<DIV>Mar 12 18:04:41 localhost kernel: Free swap: 0kB</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: 1310720 pages of RAM</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: 305192 reserved pages</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: 12126 pages shared</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: 0 pages swap cached</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Out of memory: Killed process 4738, UID 0, (glusterfs).</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: irqbalance invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: </DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Call Trace:</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: [<ffffffff800c723e>] out_of_memory+0x8e/0x2f3</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: [<ffffffff8002e22d>] __wake_up+0x38/0x4f</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: [<ffffffff8000f53f>] __alloc_pages+0x27f/0x308</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: [<ffffffff80012eea>] __do_page_cache_readahead+0x96/0x179</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: [<ffffffff800138a2>] filemap_nopage+0x14c/0x360</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: [<ffffffff8000895e>] __handle_mm_fault+0x1fb/0x1039</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: [<ffffffff80062ff8>] thread_return+0x62/0xfe</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: [<ffffffff80066b55>] do_page_fault+0x4cb/0x874</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: [<ffffffff8005a4bc>] hrtimer_cancel+0xc/0x16</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: [<ffffffff80063d05>] do_nanosleep+0x47/0x70</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: [<ffffffff8005a3a9>] hrtimer_nanosleep+0x58/0x118</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: </DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Mem-info:</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Node 0 DMA per-cpu:</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 0 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 0 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 1 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 1 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 2 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 2 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 3 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 3 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 4 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 4 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 5 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 5 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 6 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 6 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 7 hot: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 7 cold: high 0, batch 1 used:0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Node 0 DMA32 per-cpu:</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 0 hot: high 186, batch 31 used:183</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 0 cold: high 62, batch 15 used:49</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 1 hot: high 186, batch 31 used:20</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 1 cold: high 62, batch 15 used:54</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 2 hot: high 186, batch 31 used:36</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 2 cold: high 62, batch 15 used:23</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 3 hot: high 186, batch 31 used:113</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 3 cold: high 62, batch 15 used:40</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 4 hot: high 186, batch 31 used:21</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 4 cold: high 62, batch 15 used:53</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 5 hot: high 186, batch 31 used:24</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 5 cold: high 62, batch 15 used:51</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 6 hot: high 186, batch 31 used:44</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 6 cold: high 62, batch 15 used:49</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 7 hot: high 186, batch 31 used:32</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 7 cold: high 62, batch 15 used:52</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Node 0 Normal per-cpu:</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 0 hot: high 186, batch 31 used:127</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 0 cold: high 62, batch 15 used:43</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 1 hot: high 186, batch 31 used:42</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 1 cold: high 62, batch 15 used:56</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 2 hot: high 186, batch 31 used:38</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 2 cold: high 62, batch 15 used:42</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 3 hot: high 186, batch 31 used:31</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 3 cold: high 62, batch 15 used:43</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 4 hot: high 186, batch 31 used:142</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 4 cold: high 62, batch 15 used:51</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 5 hot: high 186, batch 31 used:9</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 5 cold: high 62, batch 15 used:51</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 6 hot: high 186, batch 31 used:50</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 6 cold: high 62, batch 15 used:55</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 7 hot: high 186, batch 31 used:49</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: cpu 7 cold: high 62, batch 15 used:57</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Node 0 HighMem per-cpu: empty</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Free pages: 47324kB (0kB HighMem)</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Active:941925 inactive:356 dirty:0 writeback:0 unstable:0 free:11831 slab:39893 mapped-file:1 mapped-anon:938241 pagetables:3947</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Node 0 DMA free:10876kB min:80kB low:100kB high:120kB active:0kB inactive:0kB present:10476kB pages_scanned:0 all_unreclaimable? yes</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: lowmem_reserve[]: 0 2978 3988 3988</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Node 0 DMA32 free:28568kB min:24404kB low:30504kB high:36604kB active:2963940kB inactive:48kB present:3049956kB pages_scanned:494718 all_unreclaimable? no</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: lowmem_reserve[]: 0 0 1010 1010</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Node 0 Normal free:7880kB min:8276kB low:10344kB high:12412kB active:803444kB inactive:1420kB present:1034240kB pages_scanned:6454043 all_unreclaimable? yes</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: lowmem_reserve[]: 0 0 0 0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: lowmem_reserve[]: 0 0 0 0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Node 0 DMA: 3*4kB 0*8kB 5*16kB 3*32kB 5*64kB 3*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 10876kB</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Node 0 DMA32: 1*4kB 9*8kB 3*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 6*4096kB = 28508kB</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Node 0 Normal: 10*4kB 6*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 7880kB</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Node 0 HighMem: empty</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: 4366 pagecache pages</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Free swap = 0kB</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Total swap = 0kB</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Free swap: 0kB</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: 1310720 pages of RAM</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: 305192 reserved pages</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: 12225 pages shared</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: 0 pages swap cached</DIV>
<DIV>Mar 12 18:04:42 localhost kernel: Out of memory: Killed process 4741, UID 0, (glusterfs).</DIV>
<DIV>Mar 12 19:26:52 localhost GlusterFS[4743]: [2013-03-12 19:26:52.844286] C [rpc-clnt.c:476:rpc_clnt_fill_request_info] 0-tcfstest-client-0: cannot lookup the saved frame corresponding to xid (74) </DIV>
<DIV>Mar 12 23:50:01 localhost syslogd 1.4.1: restart (remote reception).</DIV>
<DIV>Mar 13 09:18:01 localhost auditd[2895]: Audit daemon rotating log files</DIV>
<DIV>Mar 13 09:53:49 localhost rpc.statd[6821]: Version 1.0.9 Starting</DIV>
<DIV>Mar 13 09:53:49 localhost rpc.statd[6821]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user </DIV>
<DIV>Mar 13 09:56:40 localhost kernel: fuse init (API version 7.10)</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>log from nfs server in /var/log/glusterfs/nfs.log:</DIV>
<DIV>[2013-03-13 09:53:48.901714] I [glusterfsd.c:1666:main] 0-/sbin/glusterfs: Started running /sbin/glusterfs version 3.3.0</DIV>
<DIV>[2013-03-13 09:53:49.031612] I [nfs.c:821:init] 0-nfs: NFS service started</DIV>
<DIV>[2013-03-13 09:53:49.039195] W [graph.c:316:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized</DIV>
<DIV>[2013-03-13 09:53:49.039238] W [graph.c:316:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized</DIV>
<DIV>[2013-03-13 09:53:49.039258] W [graph.c:316:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized</DIV>
<DIV>[2013-03-13 09:53:49.039304] I [client.c:2142:notify] 0-tcfstest-client-0: parent translators are ready, attempting connect on transport</DIV>
<DIV>[2013-03-13 09:53:49.044451] I [client.c:2142:notify] 0-tcfstest-client-1: parent translators are ready, attempting connect on transport</DIV>
<DIV>[2013-03-13 09:53:49.047895] I [client.c:2142:notify] 0-tcfstest-client-2: parent translators are ready, attempting connect on transport</DIV>
<DIV>[2013-03-13 09:53:49.051169] I [client.c:2142:notify] 0-tcfstest-client-3: parent translators are ready, attempting connect on transport</DIV>
<DIV>[2013-03-13 09:53:49.054372] I [client.c:2142:notify] 0-tcfstest-client-4: parent translators are ready, attempting connect on transport</DIV>
<DIV>[2013-03-13 09:53:49.057610] I [client.c:2142:notify] 0-tcfstest-client-5: parent translators are ready, attempting connect on transport</DIV>
<DIV>Given volfile:</DIV>
<DIV>+------------------------------------------------------------------------------+</DIV>
<DIV> 1: volume tcfstest-client-0</DIV>
<DIV> 2: type protocol/client</DIV>
<DIV> 3: option remote-host 125.210.140.17</DIV>
<DIV> 4: option remote-subvolume /mnt/p1/exp</DIV>
<DIV> 5: option transport-type tcp</DIV>
<DIV> 6: option username 51b818d5-9b20-402a-9087-b03c5a91b01f</DIV>
<DIV> 7: option password fc720fbe-23b7-4696-971f-09d57c822e82</DIV>
<DIV> 8: end-volume</DIV>
<DIV> 9: </DIV>
<DIV> 10: volume tcfstest-client-1</DIV>
<DIV> 11: type protocol/client</DIV>
<DIV> 12: option remote-host 125.210.140.18</DIV>
<DIV> 13: option remote-subvolume /mnt/p1/exp</DIV>
<DIV> 14: option transport-type tcp</DIV>
<DIV> 15: option username 51b818d5-9b20-402a-9087-b03c5a91b01f</DIV>
<DIV> 16: option password fc720fbe-23b7-4696-971f-09d57c822e82</DIV>
<DIV> 17: end-volume</DIV>
<DIV> 18: </DIV>
<DIV> 19: volume tcfstest-client-2</DIV>
<DIV> 20: type protocol/client</DIV>
<DIV> 21: option remote-host 125.210.140.17</DIV>
<DIV> 22: option remote-subvolume /mnt/p2/exp</DIV>
<DIV> 23: option transport-type tcp</DIV>
<DIV> 24: option username 51b818d5-9b20-402a-9087-b03c5a91b01f</DIV>
<DIV> 25: option password fc720fbe-23b7-4696-971f-09d57c822e82</DIV>
<DIV> 26: end-volume</DIV>
<DIV> 27: </DIV>
<DIV> 28: volume tcfstest-client-3</DIV>
<DIV> 29: type protocol/client</DIV>
<DIV> 30: option remote-host 125.210.140.18</DIV>
<DIV> 31: option remote-subvolume /mnt/p2/exp</DIV>
<DIV> 32: option transport-type tcp</DIV>
<DIV> 33: option username 51b818d5-9b20-402a-9087-b03c5a91b01f</DIV>
<DIV> 34: option password fc720fbe-23b7-4696-971f-09d57c822e82</DIV>
<DIV> 35: end-volume</DIV>
<DIV> 36: </DIV>
<DIV> 37: volume tcfstest-client-4</DIV>
<DIV> 38: type protocol/client</DIV>
<DIV> 39: option remote-host 125.210.140.19</DIV>
<DIV> 40: option remote-subvolume /mnt/p1/exp</DIV>
<DIV> 41: option transport-type tcp</DIV>
<DIV> 42: option username 51b818d5-9b20-402a-9087-b03c5a91b01f</DIV>
<DIV> 43: option password fc720fbe-23b7-4696-971f-09d57c822e82</DIV>
<DIV> 44: end-volume</DIV>
<DIV> 45: </DIV>
<DIV> 46: volume tcfstest-client-5</DIV>
<DIV> 47: type protocol/client</DIV>
<DIV> 48: option remote-host 125.210.140.20</DIV>
<DIV> 49: option remote-subvolume /mnt/p1/exp</DIV>
<DIV> 50: option transport-type tcp</DIV>
<DIV> 51: option username 51b818d5-9b20-402a-9087-b03c5a91b01f</DIV>
<DIV> 52: option password fc720fbe-23b7-4696-971f-09d57c822e82</DIV>
<DIV> 53: end-volume</DIV>
<DIV> 54: </DIV>
<DIV> 55: volume tcfstest-replicate-0</DIV>
<DIV> 56: type cluster/replicate</DIV>
<DIV> 57: subvolumes tcfstest-client-0 tcfstest-client-1</DIV>
<DIV> 58: end-volume</DIV>
<DIV> 59: </DIV>
<DIV> 60: volume tcfstest-replicate-1</DIV>
<DIV> 61: type cluster/replicate</DIV>
<DIV> 62: subvolumes tcfstest-client-2 tcfstest-client-3</DIV>
<DIV> 63: end-volume</DIV>
<DIV> 64: </DIV>
<DIV> 65: volume tcfstest-replicate-2</DIV>
<DIV> 66: type cluster/replicate</DIV>
<DIV> 67: subvolumes tcfstest-client-4 tcfstest-client-5</DIV>
<DIV> 68: end-volume</DIV>
<DIV> 69: </DIV>
<DIV> 70: volume tcfstest-dht</DIV>
<DIV> 71: type cluster/distribute</DIV>
<DIV> 72: subvolumes tcfstest-replicate-0 tcfstest-replicate-1 tcfstest-replicate-2</DIV>
<DIV> 73: end-volume</DIV>
<DIV> 74: </DIV>
<DIV> 75: volume tcfstest</DIV>
<DIV> 76: type debug/io-stats</DIV>
<DIV> 77: option latency-measurement off</DIV>
<DIV> 78: option count-fop-hits off</DIV>
<DIV> 79: subvolumes tcfstest-dht</DIV>
<DIV> 80: end-volume</DIV>
<DIV> 81: </DIV>
<DIV> 82: volume nfs-server</DIV>
<DIV> 83: type nfs/server</DIV>
<DIV> 84: option nfs.dynamic-volumes on</DIV>
<DIV> 85: option nfs.nlm on</DIV>
<DIV> 86: option rpc-auth.addr.tcfstest.allow *</DIV>
<DIV> 87: option nfs3.tcfstest.volume-id a880c23b-b02b-4bb6-94b4-80829a893a20</DIV>
<DIV> 88: subvolumes tcfstest</DIV>
<DIV> 89: end-volume</DIV>
<DIV> </DIV>
<DIV>+------------------------------------------------------------------------------+</DIV>
<DIV>[2013-03-13 09:53:49.061446] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-tcfstest-client-2: changing port to 24014 (from 0)</DIV>
<DIV>[2013-03-13 09:53:49.061502] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-tcfstest-client-0: changing port to 24013 (from 0)</DIV>
<DIV>[2013-03-13 09:53:49.061558] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-tcfstest-client-3: changing port to 24014 (from 0)</DIV>
<DIV>[2013-03-13 09:53:49.061615] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-tcfstest-client-1: changing port to 24013 (from 0)</DIV>
<DIV>[2013-03-13 09:53:49.061671] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-tcfstest-client-5: changing port to 24011 (from 0)</DIV>
<DIV>[2013-03-13 09:53:49.061701] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-tcfstest-client-4: changing port to 24011 (from 0)</DIV>
<DIV>[2013-03-13 09:53:51.603169] W [socket.c:410:__socket_keepalive] 0-socket: failed to set keep idle on socket 8</DIV>
<DIV>[2013-03-13 09:53:51.603224] W [socket.c:1876:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported</DIV>
<DIV>[2013-03-13 09:53:52.922575] I [client-handshake.c:1636:select_server_supported_programs] 0-tcfstest-client-2: Using Program GlusterFS 3.3.0, Num (1298437), Version (330)</DIV>
<DIV>[2013-03-13 09:53:52.925650] I [client-handshake.c:1433:client_setvolume_cbk] 0-tcfstest-client-2: Connected to 125.210.140.17:24014, attached to remote volume '/mnt/p2/exp'.</DIV>
<DIV>[2013-03-13 09:53:52.925689] I [client-handshake.c:1445:client_setvolume_cbk] 0-tcfstest-client-2: Server and Client lk-version numbers are not same, reopening the fds</DIV>
<DIV>[2013-03-13 09:53:52.925824] I [afr-common.c:3627:afr_notify] 0-tcfstest-replicate-1: Subvolume 'tcfstest-client-2' came back up; going online.</DIV>
<DIV>[2013-03-13 09:53:52.926002] I [client-handshake.c:453:client_set_lk_version_cbk] 0-tcfstest-client-2: Server lk version = 1</DIV>
<DIV>[2013-03-13 09:53:52.926088] I [client-handshake.c:1636:select_server_supported_programs] 0-tcfstest-client-0: Using Program GlusterFS 3.3.0, Num (1298437), Version (330)</DIV>
<DIV>[2013-03-13 09:53:52.929945] I [client-handshake.c:1433:client_setvolume_cbk] 0-tcfstest-client-0: Connected to 125.210.140.17:24013, attached to remote volume '/mnt/p1/exp'.</DIV>
<DIV>[2013-03-13 09:53:52.929983] I [client-handshake.c:1445:client_setvolume_cbk] 0-tcfstest-client-0: Server and Client lk-version numbers are not same, reopening the fds</DIV>
<DIV>[2013-03-13 09:53:52.930042] I [afr-common.c:3627:afr_notify] 0-tcfstest-replicate-0: Subvolume 'tcfstest-client-0' came back up; going online.</DIV>
<DIV>[2013-03-13 09:53:52.930073] I [client-handshake.c:453:client_set_lk_version_cbk] 0-tcfstest-client-0: Server lk version = 1</DIV>
<DIV>[2013-03-13 09:53:52.930979] I [client-handshake.c:1636:select_server_supported_programs] 0-tcfstest-client-3: Using Program GlusterFS 3.3.0, Num (1298437), Version (330)</DIV>
<DIV>[2013-03-13 09:53:52.934061] I [client-handshake.c:1433:client_setvolume_cbk] 0-tcfstest-client-3: Connected to 125.210.140.18:24014, attached to remote volume '/mnt/p2/exp'.</DIV>
<DIV>[2013-03-13 09:53:52.934084] I [client-handshake.c:1445:client_setvolume_cbk] 0-tcfstest-client-3: Server and Client lk-version numbers are not same, reopening the fds</DIV>
<DIV>[2013-03-13 09:53:52.934241] I [client-handshake.c:1636:select_server_supported_programs] 0-tcfstest-client-1: Using Program GlusterFS 3.3.0, Num (1298437), Version (330)</DIV>
<DIV>[2013-03-13 09:53:52.934339] I [client-handshake.c:453:client_set_lk_version_cbk] 0-tcfstest-client-3: Server lk version = 1</DIV>
<DIV>[2013-03-13 09:53:52.937572] I [client-handshake.c:1433:client_setvolume_cbk] 0-tcfstest-client-1: Connected to 125.210.140.18:24013, attached to remote volume '/mnt/p1/exp'.</DIV>
<DIV>[2013-03-13 09:53:52.937605] I [client-handshake.c:1445:client_setvolume_cbk] 0-tcfstest-client-1: Server and Client lk-version numbers are not same, reopening the fds</DIV>
<DIV>[2013-03-13 09:53:52.937813] I [client-handshake.c:1636:select_server_supported_programs] 0-tcfstest-client-5: Using Program GlusterFS 3.3.0, Num (1298437), Version (330)</DIV>
<DIV>[2013-03-13 09:53:52.937918] I [client-handshake.c:453:client_set_lk_version_cbk] 0-tcfstest-client-1: Server lk version = 1</DIV>
<DIV>[2013-03-13 09:53:52.940899] I [client-handshake.c:1433:client_setvolume_cbk] 0-tcfstest-client-5: Connected to 125.210.140.20:24011, attached to remote volume '/mnt/p1/exp'.</DIV>
<DIV>[2013-03-13 09:53:52.940925] I [client-handshake.c:1445:client_setvolume_cbk] 0-tcfstest-client-5: Server and Client lk-version numbers are not same, reopening the fds</DIV>
<DIV>[2013-03-13 09:53:52.940972] I [afr-common.c:3627:afr_notify] 0-tcfstest-replicate-2: Subvolume 'tcfstest-client-5' came back up; going online.</DIV>
<DIV>[2013-03-13 09:53:52.941097] I [client-handshake.c:453:client_set_lk_version_cbk] 0-tcfstest-client-5: Server lk version = 1</DIV>
<DIV>[2013-03-13 09:53:52.941334] I [client-handshake.c:1636:select_server_supported_programs] 0-tcfstest-client-4: Using Program GlusterFS 3.3.0, Num (1298437), Version (330)</DIV>
<DIV>[2013-03-13 09:53:52.944357] I [client-handshake.c:1433:client_setvolume_cbk] 0-tcfstest-client-4: Connected to 125.210.140.19:24011, attached to remote volume '/mnt/p1/exp'.</DIV>
<DIV>[2013-03-13 09:53:52.944381] I [client-handshake.c:1445:client_setvolume_cbk] 0-tcfstest-client-4: Server and Client lk-version numbers are not same, reopening the fds</DIV>
<DIV>[2013-03-13 09:53:52.951156] I [client-handshake.c:453:client_set_lk_version_cbk] 0-tcfstest-client-4: Server lk version = 1</DIV>
<DIV>[2013-03-13 09:53:52.951332] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-tcfstest-replicate-0: added root inode</DIV>
<DIV>[2013-03-13 09:53:52.951889] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-tcfstest-replicate-1: added root inode</DIV>
<DIV>[2013-03-13 09:53:52.952169] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-tcfstest-replicate-2: added root inode</DIV></DIV>
<HR style="WIDTH: 210px; HEIGHT: 1px" align=left color=#b5c4df SIZE=1>
<DIV><SPAN>Pippo</SPAN></DIV>
<DIV> </DIV>
<DIV
style="BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0cm; PADDING-LEFT: 0cm; PADDING-RIGHT: 0cm; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<DIV
style="PADDING-BOTTOM: 8px; PADDING-LEFT: 8px; PADDING-RIGHT: 8px; BACKGROUND: #efefef; COLOR: #000000; FONT-SIZE: 12px; PADDING-TOP: 8px">
<DIV><B>From:</B> <A href="mailto:pkarampu@redhat.com">Pranith Kumar
K</A></DIV>
<DIV><B>Date:</B> 2013-03-26 01:31</DIV>
<DIV><B>To:</B> <A href="mailto:pippo0805@163.com">pippo0805</A></DIV>
<DIV><B>CC:</B> <A
href="mailto:gluster-users@gluster.org">gluster-users</A></DIV>
<DIV><B>Subject:</B> Re: [Gluster-users] A problem when mount glusterfs via
NFS</DIV></DIV></DIV>
<DIV>
<DIV class=FoxDiv20130326074000788998>
<DIV class=moz-cite-prefix>On 03/25/2013 08:23 AM, Pippo wrote:<BR></DIV>
<BLOCKQUOTE cite=mid:201303251053374967568@163.com type="cite">
<STYLE>BLOCKQUOTE {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px; MARGIN-LEFT: 2em
}
OL {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
UL {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
P {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
</STYLE>
<META name=GENERATOR content="MSHTML 8.00.7601.17514">
<DIV>HI:</DIV>
<DIV> </DIV>
<DIV style="TEXT-INDENT: 2em">
<DIV>I run glusterfs with four nodes, 2x2 Distributed-Replicate.</DIV>
<DIV>I mounted it via fuse and did some test, it was ok.</DIV>
<DIV>However when I mounted it via nfs, a problem was found:</DIV>
<DIV> </DIV>
<DIV style="TEXT-INDENT: 2em">
When I copied 200G files to the glusterfs, the glusterfs process in the server node(mounted by client) was killed because of OOM, </DIV>
<DIV>and all terminals of the client were hung. Trying to test for many times, I got the same result. The heavier load I pushed via the</DIV>
<DIV>client, the faster glusterfs process been killed. I run "top" in the server, found that the glusterfs process eat MEM very fast,
and </DIV>
<DIV>never gone down until it was killed. I think it is a bug of glusterfs process, it leak memory.</DIV>
<DIV> </DIV>
<DIV>I google "glusterfs OOM" but can not find any solutions. Is anyone know
about this problem and give me some tips? Many thanks!</DIV></DIV>
<DIV> </DIV>
<HR style="WIDTH: 210px; HEIGHT: 1px" align=left color=#b5c4df SIZE=1>
<DIV><SPAN>Pippo</SPAN></DIV><BR>
<FIELDSET class=mimeAttachmentHeader></FIELDSET> <BR><PRE wrap="">_______________________________________________
Gluster-users mailing list
<A class=moz-txt-link-abbreviated href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</A>
<A class=moz-txt-link-freetext href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</A></PRE></BLOCKQUOTE>hi,<BR>
Could you let us know the version of glusterfs you were using? nfs server Logs
of that run would help us if you could attach that to this
mail.<BR><BR>Pranith<BR></DIV></DIV></BODY></HTML>