<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=gb2312" http-equiv=Content-Type>
<STYLE>
BLOCKQUOTE {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px; MARGIN-LEFT: 2em
}
OL {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
UL {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
DIV.FoxDiv20130326074000788998 {
        LINE-HEIGHT: 1.5; FONT-FAMILY: &amp; COLOR: #000000; FONT-SIZE: 10.5pt; 24494: ; amp: ; 36719: ; 38597: ; 40657: 
}
BODY {
        LINE-HEIGHT: 1.5; FONT-FAMILY: Î¢ÈíÑźÚ; COLOR: #000080; FONT-SIZE: 10.5pt
}
</STYLE>

<META name=GENERATOR content="MSHTML 8.00.7601.17514"></HEAD>
<BODY style="MARGIN: 10px" bgColor=#ffffff text=#000000>
<DIV>
<DIV>Hi&nbsp;Pranith:</DIV>
<DIV>&nbsp;</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;Thanks&nbsp;for&nbsp;your&nbsp;reply.&nbsp;I&nbsp;run&nbsp;Glusterfs&nbsp;3.3¡£</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;I&nbsp;have&nbsp;a&nbsp;very&nbsp;important&nbsp;situation&nbsp;forgot&nbsp;to&nbsp;describe:</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;When&nbsp;the&nbsp;nfs&nbsp;server&nbsp;and&nbsp;client&nbsp;are&nbsp;both&nbsp;running&nbsp;CentOS&nbsp;5.5,they&nbsp;work&nbsp;well.&nbsp;The&nbsp;problem&nbsp;appear&nbsp;only&nbsp;the&nbsp;nfs&nbsp;server&nbsp;run&nbsp;CentOS&nbsp;5.5&nbsp;and&nbsp;the 
client&nbsp;run&nbsp;CentOS&nbsp;6.3.</DIV>
<DIV>&nbsp;&nbsp;&nbsp;&nbsp;</DIV>
<DIV>The&nbsp;/var/log/messages:&nbsp;</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;glusterfs&nbsp;invoked&nbsp;oom-killer:&nbsp;gfp_mask=0x280d2,&nbsp;order=0,&nbsp;oomkilladj=0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Call&nbsp;Trace:</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff800c723e&gt;]&nbsp;out_of_memory+0x8e/0x2f3</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff8002e22d&gt;]&nbsp;__wake_up+0x38/0x4f</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff8000f53f&gt;]&nbsp;__alloc_pages+0x27f/0x308</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff80008e9f&gt;]&nbsp;__handle_mm_fault+0x73c/0x1039</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff80066b55&gt;]&nbsp;do_page_fault+0x4cb/0x874</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff800f8935&gt;]&nbsp;sys_epoll_wait+0x3b8/0x3f9</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff8005dde9&gt;]&nbsp;error_exit+0x0/0x84</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Mem-info:</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;DMA&nbsp;per-cpu:</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;0&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;0&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;1&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;1&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;2&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;2&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;3&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;3&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;4&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;4&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;5&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;5&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;6&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;6&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;7&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;7&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;DMA32&nbsp;per-cpu:</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;0&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:131</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;0&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:49</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;1&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:13</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;1&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:54</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;2&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:30</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;2&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:23</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;3&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:106</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;3&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:40</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;4&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:19</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;4&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:52</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;5&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:19</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;5&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:51</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;6&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:38</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;6&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:49</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;7&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:27</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;7&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:48</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;Normal&nbsp;per-cpu:</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;0&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:59</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;0&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:43</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;1&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:36</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;1&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:56</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;2&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:25</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;2&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:42</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;3&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:22</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;3&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:43</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;4&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:140</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;4&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:51</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;5&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:2</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;5&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:51</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;6&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:39</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;6&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:55</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;7&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:28</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;7&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:57</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;HighMem&nbsp;per-cpu:&nbsp;empty</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Free&nbsp;pages:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;47116kB&nbsp;(0kB&nbsp;HighMem)</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Active:941872&nbsp;inactive:355&nbsp;dirty:0&nbsp;writeback:0&nbsp;unstable:0&nbsp;free:11779&nbsp;slab:40203&nbsp;mapped-file:1&nbsp;mapped-anon:938241&nbsp;pagetables:3947</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;DMA&nbsp;free:10876kB&nbsp;min:80kB&nbsp;low:100kB&nbsp;high:120kB&nbsp;active:0kB&nbsp;inactive:0kB&nbsp;present:10476kB&nbsp;pages_scanned:0&nbsp;all_unreclaimable?&nbsp;yes</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;lowmem_reserve[]:&nbsp;0&nbsp;2978&nbsp;3988&nbsp;3988</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;DMA32&nbsp;free:28400kB&nbsp;min:24404kB&nbsp;low:30504kB&nbsp;high:36604kB&nbsp;active:2964256kB&nbsp;inactive:0kB&nbsp;present:3049956kB&nbsp;pages_scanned:15320246&nbsp;all_unreclaimable?&nbsp;yes</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;lowmem_reserve[]:&nbsp;0&nbsp;0&nbsp;1010&nbsp;1010</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;Normal&nbsp;free:7840kB&nbsp;min:8276kB&nbsp;low:10344kB&nbsp;high:12412kB&nbsp;active:803516kB&nbsp;inactive:1292kB&nbsp;present:1034240kB&nbsp;pages_scanned:7795789&nbsp;all_unreclaimable?&nbsp;yes</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;lowmem_reserve[]:&nbsp;0&nbsp;0&nbsp;0&nbsp;0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;HighMem&nbsp;free:0kB&nbsp;min:128kB&nbsp;low:128kB&nbsp;high:128kB&nbsp;active:0kB&nbsp;inactive:0kB&nbsp;present:0kB&nbsp;pages_scanned:0&nbsp;all_unreclaimable?&nbsp;no</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;lowmem_reserve[]:&nbsp;0&nbsp;0&nbsp;0&nbsp;0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;DMA:&nbsp;3*4kB&nbsp;0*8kB&nbsp;5*16kB&nbsp;3*32kB&nbsp;5*64kB&nbsp;3*128kB&nbsp;1*256kB&nbsp;1*512kB&nbsp;1*1024kB&nbsp;0*2048kB&nbsp;2*4096kB&nbsp;=&nbsp;10876kB</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;DMA32:&nbsp;16*4kB&nbsp;0*8kB&nbsp;1*16kB&nbsp;1*32kB&nbsp;0*64kB&nbsp;1*128kB&nbsp;0*256kB&nbsp;1*512kB&nbsp;1*1024kB&nbsp;1*2048kB&nbsp;6*4096kB&nbsp;=&nbsp;28400kB</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;Normal:&nbsp;10*4kB&nbsp;3*8kB&nbsp;0*16kB&nbsp;1*32kB&nbsp;1*64kB&nbsp;0*128kB&nbsp;0*256kB&nbsp;1*512kB&nbsp;1*1024kB&nbsp;1*2048kB&nbsp;1*4096kB&nbsp;=&nbsp;7840kB</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;HighMem:&nbsp;empty</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;4366&nbsp;pagecache&nbsp;pages</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Swap&nbsp;cache:&nbsp;add&nbsp;0,&nbsp;delete&nbsp;0,&nbsp;find&nbsp;0/0,&nbsp;race&nbsp;0+0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Free&nbsp;swap&nbsp;&nbsp;=&nbsp;0kB</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Total&nbsp;swap&nbsp;=&nbsp;0kB</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:41&nbsp;localhost&nbsp;kernel:&nbsp;Free&nbsp;swap:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0kB</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;1310720&nbsp;pages&nbsp;of&nbsp;RAM</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;305192&nbsp;reserved&nbsp;pages</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;12126&nbsp;pages&nbsp;shared</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;0&nbsp;pages&nbsp;swap&nbsp;cached</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Out&nbsp;of&nbsp;memory:&nbsp;Killed&nbsp;process&nbsp;4738,&nbsp;UID&nbsp;0,&nbsp;(glusterfs).</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;irqbalance&nbsp;invoked&nbsp;oom-killer:&nbsp;gfp_mask=0x201d2,&nbsp;order=0,&nbsp;oomkilladj=0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Call&nbsp;Trace:</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff800c723e&gt;]&nbsp;out_of_memory+0x8e/0x2f3</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff8002e22d&gt;]&nbsp;__wake_up+0x38/0x4f</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff8000f53f&gt;]&nbsp;__alloc_pages+0x27f/0x308</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff80012eea&gt;]&nbsp;__do_page_cache_readahead+0x96/0x179</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff800138a2&gt;]&nbsp;filemap_nopage+0x14c/0x360</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff8000895e&gt;]&nbsp;__handle_mm_fault+0x1fb/0x1039</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff80062ff8&gt;]&nbsp;thread_return+0x62/0xfe</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff80066b55&gt;]&nbsp;do_page_fault+0x4cb/0x874</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff8005a4bc&gt;]&nbsp;hrtimer_cancel+0xc/0x16</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff80063d05&gt;]&nbsp;do_nanosleep+0x47/0x70</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff8005a3a9&gt;]&nbsp;hrtimer_nanosleep+0x58/0x118</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;&nbsp;[&lt;ffffffff8005dde9&gt;]&nbsp;error_exit+0x0/0x84</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Mem-info:</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;DMA&nbsp;per-cpu:</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;0&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;0&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;1&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;1&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;2&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;2&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;3&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;3&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;4&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;4&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;5&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;5&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;6&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;6&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;7&nbsp;hot:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;7&nbsp;cold:&nbsp;high&nbsp;0,&nbsp;batch&nbsp;1&nbsp;used:0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;DMA32&nbsp;per-cpu:</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;0&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:183</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;0&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:49</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;1&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:20</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;1&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:54</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;2&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:36</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;2&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:23</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;3&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:113</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;3&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:40</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;4&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:21</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;4&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:53</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;5&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:24</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;5&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:51</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;6&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:44</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;6&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:49</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;7&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:32</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;7&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:52</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;Normal&nbsp;per-cpu:</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;0&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:127</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;0&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:43</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;1&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:42</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;1&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:56</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;2&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:38</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;2&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:42</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;3&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:31</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;3&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:43</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;4&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:142</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;4&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:51</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;5&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:9</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;5&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:51</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;6&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:50</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;6&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:55</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;7&nbsp;hot:&nbsp;high&nbsp;186,&nbsp;batch&nbsp;31&nbsp;used:49</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;cpu&nbsp;7&nbsp;cold:&nbsp;high&nbsp;62,&nbsp;batch&nbsp;15&nbsp;used:57</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;HighMem&nbsp;per-cpu:&nbsp;empty</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Free&nbsp;pages:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;47324kB&nbsp;(0kB&nbsp;HighMem)</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Active:941925&nbsp;inactive:356&nbsp;dirty:0&nbsp;writeback:0&nbsp;unstable:0&nbsp;free:11831&nbsp;slab:39893&nbsp;mapped-file:1&nbsp;mapped-anon:938241&nbsp;pagetables:3947</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;DMA&nbsp;free:10876kB&nbsp;min:80kB&nbsp;low:100kB&nbsp;high:120kB&nbsp;active:0kB&nbsp;inactive:0kB&nbsp;present:10476kB&nbsp;pages_scanned:0&nbsp;all_unreclaimable?&nbsp;yes</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;lowmem_reserve[]:&nbsp;0&nbsp;2978&nbsp;3988&nbsp;3988</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;DMA32&nbsp;free:28568kB&nbsp;min:24404kB&nbsp;low:30504kB&nbsp;high:36604kB&nbsp;active:2963940kB&nbsp;inactive:48kB&nbsp;present:3049956kB&nbsp;pages_scanned:494718&nbsp;all_unreclaimable?&nbsp;no</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;lowmem_reserve[]:&nbsp;0&nbsp;0&nbsp;1010&nbsp;1010</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;Normal&nbsp;free:7880kB&nbsp;min:8276kB&nbsp;low:10344kB&nbsp;high:12412kB&nbsp;active:803444kB&nbsp;inactive:1420kB&nbsp;present:1034240kB&nbsp;pages_scanned:6454043&nbsp;all_unreclaimable?&nbsp;yes</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;lowmem_reserve[]:&nbsp;0&nbsp;0&nbsp;0&nbsp;0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;HighMem&nbsp;free:0kB&nbsp;min:128kB&nbsp;low:128kB&nbsp;high:128kB&nbsp;active:0kB&nbsp;inactive:0kB&nbsp;present:0kB&nbsp;pages_scanned:0&nbsp;all_unreclaimable?&nbsp;no</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;lowmem_reserve[]:&nbsp;0&nbsp;0&nbsp;0&nbsp;0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;DMA:&nbsp;3*4kB&nbsp;0*8kB&nbsp;5*16kB&nbsp;3*32kB&nbsp;5*64kB&nbsp;3*128kB&nbsp;1*256kB&nbsp;1*512kB&nbsp;1*1024kB&nbsp;0*2048kB&nbsp;2*4096kB&nbsp;=&nbsp;10876kB</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;DMA32:&nbsp;1*4kB&nbsp;9*8kB&nbsp;3*16kB&nbsp;1*32kB&nbsp;1*64kB&nbsp;1*128kB&nbsp;0*256kB&nbsp;1*512kB&nbsp;1*1024kB&nbsp;1*2048kB&nbsp;6*4096kB&nbsp;=&nbsp;28508kB</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;Normal:&nbsp;10*4kB&nbsp;6*8kB&nbsp;1*16kB&nbsp;1*32kB&nbsp;1*64kB&nbsp;0*128kB&nbsp;0*256kB&nbsp;1*512kB&nbsp;1*1024kB&nbsp;1*2048kB&nbsp;1*4096kB&nbsp;=&nbsp;7880kB</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Node&nbsp;0&nbsp;HighMem:&nbsp;empty</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;4366&nbsp;pagecache&nbsp;pages</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Swap&nbsp;cache:&nbsp;add&nbsp;0,&nbsp;delete&nbsp;0,&nbsp;find&nbsp;0/0,&nbsp;race&nbsp;0+0</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Free&nbsp;swap&nbsp;&nbsp;=&nbsp;0kB</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Total&nbsp;swap&nbsp;=&nbsp;0kB</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Free&nbsp;swap:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0kB</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;1310720&nbsp;pages&nbsp;of&nbsp;RAM</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;305192&nbsp;reserved&nbsp;pages</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;12225&nbsp;pages&nbsp;shared</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;0&nbsp;pages&nbsp;swap&nbsp;cached</DIV>
<DIV>Mar&nbsp;12&nbsp;18:04:42&nbsp;localhost&nbsp;kernel:&nbsp;Out&nbsp;of&nbsp;memory:&nbsp;Killed&nbsp;process&nbsp;4741,&nbsp;UID&nbsp;0,&nbsp;(glusterfs).</DIV>
<DIV>Mar&nbsp;12&nbsp;19:26:52&nbsp;localhost&nbsp;GlusterFS[4743]:&nbsp;[2013-03-12&nbsp;19:26:52.844286]&nbsp;C&nbsp;[rpc-clnt.c:476:rpc_clnt_fill_request_info]&nbsp;0-tcfstest-client-0:&nbsp;cannot&nbsp;lookup&nbsp;the&nbsp;saved&nbsp;frame&nbsp;corresponding&nbsp;to&nbsp;xid&nbsp;(74)&nbsp;</DIV>
<DIV>Mar&nbsp;12&nbsp;23:50:01&nbsp;localhost&nbsp;syslogd&nbsp;1.4.1:&nbsp;restart&nbsp;(remote&nbsp;reception).</DIV>
<DIV>Mar&nbsp;13&nbsp;09:18:01&nbsp;localhost&nbsp;auditd[2895]:&nbsp;Audit&nbsp;daemon&nbsp;rotating&nbsp;log&nbsp;files</DIV>
<DIV>Mar&nbsp;13&nbsp;09:53:49&nbsp;localhost&nbsp;rpc.statd[6821]:&nbsp;Version&nbsp;1.0.9&nbsp;Starting</DIV>
<DIV>Mar&nbsp;13&nbsp;09:53:49&nbsp;localhost&nbsp;rpc.statd[6821]:&nbsp;statd&nbsp;running&nbsp;as&nbsp;root.&nbsp;chown&nbsp;/var/lib/nfs/statd/sm&nbsp;to&nbsp;choose&nbsp;different&nbsp;user&nbsp;</DIV>
<DIV>Mar&nbsp;13&nbsp;09:56:40&nbsp;localhost&nbsp;kernel:&nbsp;fuse&nbsp;init&nbsp;(API&nbsp;version&nbsp;7.10)</DIV>
<DIV>&nbsp;</DIV>
<DIV>&nbsp;</DIV>
<DIV>log&nbsp;from&nbsp;nfs&nbsp;server&nbsp;in&nbsp;/var/log/glusterfs/nfs.log:</DIV>
<DIV>[2013-03-13&nbsp;09:53:48.901714]&nbsp;I&nbsp;[glusterfsd.c:1666:main]&nbsp;0-/sbin/glusterfs:&nbsp;Started&nbsp;running&nbsp;/sbin/glusterfs&nbsp;version&nbsp;3.3.0</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.031612]&nbsp;I&nbsp;[nfs.c:821:init]&nbsp;0-nfs:&nbsp;NFS&nbsp;service&nbsp;started</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.039195]&nbsp;W&nbsp;[graph.c:316:_log_if_unknown_option]&nbsp;0-nfs-server:&nbsp;option&nbsp;'rpc-auth.auth-glusterfs'&nbsp;is&nbsp;not&nbsp;recognized</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.039238]&nbsp;W&nbsp;[graph.c:316:_log_if_unknown_option]&nbsp;0-nfs-server:&nbsp;option&nbsp;'rpc-auth-allow-insecure'&nbsp;is&nbsp;not&nbsp;recognized</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.039258]&nbsp;W&nbsp;[graph.c:316:_log_if_unknown_option]&nbsp;0-nfs-server:&nbsp;option&nbsp;'transport-type'&nbsp;is&nbsp;not&nbsp;recognized</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.039304]&nbsp;I&nbsp;[client.c:2142:notify]&nbsp;0-tcfstest-client-0:&nbsp;parent&nbsp;translators&nbsp;are&nbsp;ready,&nbsp;attempting&nbsp;connect&nbsp;on&nbsp;transport</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.044451]&nbsp;I&nbsp;[client.c:2142:notify]&nbsp;0-tcfstest-client-1:&nbsp;parent&nbsp;translators&nbsp;are&nbsp;ready,&nbsp;attempting&nbsp;connect&nbsp;on&nbsp;transport</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.047895]&nbsp;I&nbsp;[client.c:2142:notify]&nbsp;0-tcfstest-client-2:&nbsp;parent&nbsp;translators&nbsp;are&nbsp;ready,&nbsp;attempting&nbsp;connect&nbsp;on&nbsp;transport</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.051169]&nbsp;I&nbsp;[client.c:2142:notify]&nbsp;0-tcfstest-client-3:&nbsp;parent&nbsp;translators&nbsp;are&nbsp;ready,&nbsp;attempting&nbsp;connect&nbsp;on&nbsp;transport</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.054372]&nbsp;I&nbsp;[client.c:2142:notify]&nbsp;0-tcfstest-client-4:&nbsp;parent&nbsp;translators&nbsp;are&nbsp;ready,&nbsp;attempting&nbsp;connect&nbsp;on&nbsp;transport</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.057610]&nbsp;I&nbsp;[client.c:2142:notify]&nbsp;0-tcfstest-client-5:&nbsp;parent&nbsp;translators&nbsp;are&nbsp;ready,&nbsp;attempting&nbsp;connect&nbsp;on&nbsp;transport</DIV>
<DIV>Given&nbsp;volfile:</DIV>
<DIV>+------------------------------------------------------------------------------+</DIV>
<DIV>&nbsp;&nbsp;1:&nbsp;volume&nbsp;tcfstest-client-0</DIV>
<DIV>&nbsp;&nbsp;2:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;protocol/client</DIV>
<DIV>&nbsp;&nbsp;3:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;remote-host&nbsp;125.210.140.17</DIV>
<DIV>&nbsp;&nbsp;4:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;remote-subvolume&nbsp;/mnt/p1/exp</DIV>
<DIV>&nbsp;&nbsp;5:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;transport-type&nbsp;tcp</DIV>
<DIV>&nbsp;&nbsp;6:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;username&nbsp;51b818d5-9b20-402a-9087-b03c5a91b01f</DIV>
<DIV>&nbsp;&nbsp;7:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;password&nbsp;fc720fbe-23b7-4696-971f-09d57c822e82</DIV>
<DIV>&nbsp;&nbsp;8:&nbsp;end-volume</DIV>
<DIV>&nbsp;&nbsp;9:&nbsp;</DIV>
<DIV>&nbsp;10:&nbsp;volume&nbsp;tcfstest-client-1</DIV>
<DIV>&nbsp;11:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;protocol/client</DIV>
<DIV>&nbsp;12:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;remote-host&nbsp;125.210.140.18</DIV>
<DIV>&nbsp;13:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;remote-subvolume&nbsp;/mnt/p1/exp</DIV>
<DIV>&nbsp;14:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;transport-type&nbsp;tcp</DIV>
<DIV>&nbsp;15:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;username&nbsp;51b818d5-9b20-402a-9087-b03c5a91b01f</DIV>
<DIV>&nbsp;16:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;password&nbsp;fc720fbe-23b7-4696-971f-09d57c822e82</DIV>
<DIV>&nbsp;17:&nbsp;end-volume</DIV>
<DIV>&nbsp;18:&nbsp;</DIV>
<DIV>&nbsp;19:&nbsp;volume&nbsp;tcfstest-client-2</DIV>
<DIV>&nbsp;20:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;protocol/client</DIV>
<DIV>&nbsp;21:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;remote-host&nbsp;125.210.140.17</DIV>
<DIV>&nbsp;22:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;remote-subvolume&nbsp;/mnt/p2/exp</DIV>
<DIV>&nbsp;23:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;transport-type&nbsp;tcp</DIV>
<DIV>&nbsp;24:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;username&nbsp;51b818d5-9b20-402a-9087-b03c5a91b01f</DIV>
<DIV>&nbsp;25:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;password&nbsp;fc720fbe-23b7-4696-971f-09d57c822e82</DIV>
<DIV>&nbsp;26:&nbsp;end-volume</DIV>
<DIV>&nbsp;27:&nbsp;</DIV>
<DIV>&nbsp;28:&nbsp;volume&nbsp;tcfstest-client-3</DIV>
<DIV>&nbsp;29:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;protocol/client</DIV>
<DIV>&nbsp;30:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;remote-host&nbsp;125.210.140.18</DIV>
<DIV>&nbsp;31:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;remote-subvolume&nbsp;/mnt/p2/exp</DIV>
<DIV>&nbsp;32:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;transport-type&nbsp;tcp</DIV>
<DIV>&nbsp;33:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;username&nbsp;51b818d5-9b20-402a-9087-b03c5a91b01f</DIV>
<DIV>&nbsp;34:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;password&nbsp;fc720fbe-23b7-4696-971f-09d57c822e82</DIV>
<DIV>&nbsp;35:&nbsp;end-volume</DIV>
<DIV>&nbsp;36:&nbsp;</DIV>
<DIV>&nbsp;37:&nbsp;volume&nbsp;tcfstest-client-4</DIV>
<DIV>&nbsp;38:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;protocol/client</DIV>
<DIV>&nbsp;39:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;remote-host&nbsp;125.210.140.19</DIV>
<DIV>&nbsp;40:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;remote-subvolume&nbsp;/mnt/p1/exp</DIV>
<DIV>&nbsp;41:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;transport-type&nbsp;tcp</DIV>
<DIV>&nbsp;42:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;username&nbsp;51b818d5-9b20-402a-9087-b03c5a91b01f</DIV>
<DIV>&nbsp;43:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;password&nbsp;fc720fbe-23b7-4696-971f-09d57c822e82</DIV>
<DIV>&nbsp;44:&nbsp;end-volume</DIV>
<DIV>&nbsp;45:&nbsp;</DIV>
<DIV>&nbsp;46:&nbsp;volume&nbsp;tcfstest-client-5</DIV>
<DIV>&nbsp;47:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;protocol/client</DIV>
<DIV>&nbsp;48:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;remote-host&nbsp;125.210.140.20</DIV>
<DIV>&nbsp;49:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;remote-subvolume&nbsp;/mnt/p1/exp</DIV>
<DIV>&nbsp;50:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;transport-type&nbsp;tcp</DIV>
<DIV>&nbsp;51:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;username&nbsp;51b818d5-9b20-402a-9087-b03c5a91b01f</DIV>
<DIV>&nbsp;52:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;password&nbsp;fc720fbe-23b7-4696-971f-09d57c822e82</DIV>
<DIV>&nbsp;53:&nbsp;end-volume</DIV>
<DIV>&nbsp;54:&nbsp;</DIV>
<DIV>&nbsp;55:&nbsp;volume&nbsp;tcfstest-replicate-0</DIV>
<DIV>&nbsp;56:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;cluster/replicate</DIV>
<DIV>&nbsp;57:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;subvolumes&nbsp;tcfstest-client-0&nbsp;tcfstest-client-1</DIV>
<DIV>&nbsp;58:&nbsp;end-volume</DIV>
<DIV>&nbsp;59:&nbsp;</DIV>
<DIV>&nbsp;60:&nbsp;volume&nbsp;tcfstest-replicate-1</DIV>
<DIV>&nbsp;61:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;cluster/replicate</DIV>
<DIV>&nbsp;62:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;subvolumes&nbsp;tcfstest-client-2&nbsp;tcfstest-client-3</DIV>
<DIV>&nbsp;63:&nbsp;end-volume</DIV>
<DIV>&nbsp;64:&nbsp;</DIV>
<DIV>&nbsp;65:&nbsp;volume&nbsp;tcfstest-replicate-2</DIV>
<DIV>&nbsp;66:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;cluster/replicate</DIV>
<DIV>&nbsp;67:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;subvolumes&nbsp;tcfstest-client-4&nbsp;tcfstest-client-5</DIV>
<DIV>&nbsp;68:&nbsp;end-volume</DIV>
<DIV>&nbsp;69:&nbsp;</DIV>
<DIV>&nbsp;70:&nbsp;volume&nbsp;tcfstest-dht</DIV>
<DIV>&nbsp;71:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;cluster/distribute</DIV>
<DIV>&nbsp;72:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;subvolumes&nbsp;tcfstest-replicate-0&nbsp;tcfstest-replicate-1&nbsp;tcfstest-replicate-2</DIV>
<DIV>&nbsp;73:&nbsp;end-volume</DIV>
<DIV>&nbsp;74:&nbsp;</DIV>
<DIV>&nbsp;75:&nbsp;volume&nbsp;tcfstest</DIV>
<DIV>&nbsp;76:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;debug/io-stats</DIV>
<DIV>&nbsp;77:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;latency-measurement&nbsp;off</DIV>
<DIV>&nbsp;78:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;count-fop-hits&nbsp;off</DIV>
<DIV>&nbsp;79:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;subvolumes&nbsp;tcfstest-dht</DIV>
<DIV>&nbsp;80:&nbsp;end-volume</DIV>
<DIV>&nbsp;81:&nbsp;</DIV>
<DIV>&nbsp;82:&nbsp;volume&nbsp;nfs-server</DIV>
<DIV>&nbsp;83:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;nfs/server</DIV>
<DIV>&nbsp;84:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;nfs.dynamic-volumes&nbsp;on</DIV>
<DIV>&nbsp;85:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;nfs.nlm&nbsp;on</DIV>
<DIV>&nbsp;86:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;rpc-auth.addr.tcfstest.allow&nbsp;*</DIV>
<DIV>&nbsp;87:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;option&nbsp;nfs3.tcfstest.volume-id&nbsp;a880c23b-b02b-4bb6-94b4-80829a893a20</DIV>
<DIV>&nbsp;88:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;subvolumes&nbsp;tcfstest</DIV>
<DIV>&nbsp;89:&nbsp;end-volume</DIV>
<DIV>&nbsp;</DIV>
<DIV>+------------------------------------------------------------------------------+</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.061446]&nbsp;I&nbsp;[rpc-clnt.c:1660:rpc_clnt_reconfig]&nbsp;0-tcfstest-client-2:&nbsp;changing&nbsp;port&nbsp;to&nbsp;24014&nbsp;(from&nbsp;0)</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.061502]&nbsp;I&nbsp;[rpc-clnt.c:1660:rpc_clnt_reconfig]&nbsp;0-tcfstest-client-0:&nbsp;changing&nbsp;port&nbsp;to&nbsp;24013&nbsp;(from&nbsp;0)</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.061558]&nbsp;I&nbsp;[rpc-clnt.c:1660:rpc_clnt_reconfig]&nbsp;0-tcfstest-client-3:&nbsp;changing&nbsp;port&nbsp;to&nbsp;24014&nbsp;(from&nbsp;0)</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.061615]&nbsp;I&nbsp;[rpc-clnt.c:1660:rpc_clnt_reconfig]&nbsp;0-tcfstest-client-1:&nbsp;changing&nbsp;port&nbsp;to&nbsp;24013&nbsp;(from&nbsp;0)</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.061671]&nbsp;I&nbsp;[rpc-clnt.c:1660:rpc_clnt_reconfig]&nbsp;0-tcfstest-client-5:&nbsp;changing&nbsp;port&nbsp;to&nbsp;24011&nbsp;(from&nbsp;0)</DIV>
<DIV>[2013-03-13&nbsp;09:53:49.061701]&nbsp;I&nbsp;[rpc-clnt.c:1660:rpc_clnt_reconfig]&nbsp;0-tcfstest-client-4:&nbsp;changing&nbsp;port&nbsp;to&nbsp;24011&nbsp;(from&nbsp;0)</DIV>
<DIV>[2013-03-13&nbsp;09:53:51.603169]&nbsp;W&nbsp;[socket.c:410:__socket_keepalive]&nbsp;0-socket:&nbsp;failed&nbsp;to&nbsp;set&nbsp;keep&nbsp;idle&nbsp;on&nbsp;socket&nbsp;8</DIV>
<DIV>[2013-03-13&nbsp;09:53:51.603224]&nbsp;W&nbsp;[socket.c:1876:socket_server_event_handler]&nbsp;0-socket.glusterfsd:&nbsp;Failed&nbsp;to&nbsp;set&nbsp;keep-alive:&nbsp;Operation&nbsp;not&nbsp;supported</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.922575]&nbsp;I&nbsp;[client-handshake.c:1636:select_server_supported_programs]&nbsp;0-tcfstest-client-2:&nbsp;Using&nbsp;Program&nbsp;GlusterFS&nbsp;3.3.0,&nbsp;Num&nbsp;(1298437),&nbsp;Version&nbsp;(330)</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.925650]&nbsp;I&nbsp;[client-handshake.c:1433:client_setvolume_cbk]&nbsp;0-tcfstest-client-2:&nbsp;Connected&nbsp;to&nbsp;125.210.140.17:24014,&nbsp;attached&nbsp;to&nbsp;remote&nbsp;volume&nbsp;'/mnt/p2/exp'.</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.925689]&nbsp;I&nbsp;[client-handshake.c:1445:client_setvolume_cbk]&nbsp;0-tcfstest-client-2:&nbsp;Server&nbsp;and&nbsp;Client&nbsp;lk-version&nbsp;numbers&nbsp;are&nbsp;not&nbsp;same,&nbsp;reopening&nbsp;the&nbsp;fds</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.925824]&nbsp;I&nbsp;[afr-common.c:3627:afr_notify]&nbsp;0-tcfstest-replicate-1:&nbsp;Subvolume&nbsp;'tcfstest-client-2'&nbsp;came&nbsp;back&nbsp;up;&nbsp;going&nbsp;online.</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.926002]&nbsp;I&nbsp;[client-handshake.c:453:client_set_lk_version_cbk]&nbsp;0-tcfstest-client-2:&nbsp;Server&nbsp;lk&nbsp;version&nbsp;=&nbsp;1</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.926088]&nbsp;I&nbsp;[client-handshake.c:1636:select_server_supported_programs]&nbsp;0-tcfstest-client-0:&nbsp;Using&nbsp;Program&nbsp;GlusterFS&nbsp;3.3.0,&nbsp;Num&nbsp;(1298437),&nbsp;Version&nbsp;(330)</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.929945]&nbsp;I&nbsp;[client-handshake.c:1433:client_setvolume_cbk]&nbsp;0-tcfstest-client-0:&nbsp;Connected&nbsp;to&nbsp;125.210.140.17:24013,&nbsp;attached&nbsp;to&nbsp;remote&nbsp;volume&nbsp;'/mnt/p1/exp'.</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.929983]&nbsp;I&nbsp;[client-handshake.c:1445:client_setvolume_cbk]&nbsp;0-tcfstest-client-0:&nbsp;Server&nbsp;and&nbsp;Client&nbsp;lk-version&nbsp;numbers&nbsp;are&nbsp;not&nbsp;same,&nbsp;reopening&nbsp;the&nbsp;fds</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.930042]&nbsp;I&nbsp;[afr-common.c:3627:afr_notify]&nbsp;0-tcfstest-replicate-0:&nbsp;Subvolume&nbsp;'tcfstest-client-0'&nbsp;came&nbsp;back&nbsp;up;&nbsp;going&nbsp;online.</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.930073]&nbsp;I&nbsp;[client-handshake.c:453:client_set_lk_version_cbk]&nbsp;0-tcfstest-client-0:&nbsp;Server&nbsp;lk&nbsp;version&nbsp;=&nbsp;1</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.930979]&nbsp;I&nbsp;[client-handshake.c:1636:select_server_supported_programs]&nbsp;0-tcfstest-client-3:&nbsp;Using&nbsp;Program&nbsp;GlusterFS&nbsp;3.3.0,&nbsp;Num&nbsp;(1298437),&nbsp;Version&nbsp;(330)</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.934061]&nbsp;I&nbsp;[client-handshake.c:1433:client_setvolume_cbk]&nbsp;0-tcfstest-client-3:&nbsp;Connected&nbsp;to&nbsp;125.210.140.18:24014,&nbsp;attached&nbsp;to&nbsp;remote&nbsp;volume&nbsp;'/mnt/p2/exp'.</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.934084]&nbsp;I&nbsp;[client-handshake.c:1445:client_setvolume_cbk]&nbsp;0-tcfstest-client-3:&nbsp;Server&nbsp;and&nbsp;Client&nbsp;lk-version&nbsp;numbers&nbsp;are&nbsp;not&nbsp;same,&nbsp;reopening&nbsp;the&nbsp;fds</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.934241]&nbsp;I&nbsp;[client-handshake.c:1636:select_server_supported_programs]&nbsp;0-tcfstest-client-1:&nbsp;Using&nbsp;Program&nbsp;GlusterFS&nbsp;3.3.0,&nbsp;Num&nbsp;(1298437),&nbsp;Version&nbsp;(330)</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.934339]&nbsp;I&nbsp;[client-handshake.c:453:client_set_lk_version_cbk]&nbsp;0-tcfstest-client-3:&nbsp;Server&nbsp;lk&nbsp;version&nbsp;=&nbsp;1</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.937572]&nbsp;I&nbsp;[client-handshake.c:1433:client_setvolume_cbk]&nbsp;0-tcfstest-client-1:&nbsp;Connected&nbsp;to&nbsp;125.210.140.18:24013,&nbsp;attached&nbsp;to&nbsp;remote&nbsp;volume&nbsp;'/mnt/p1/exp'.</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.937605]&nbsp;I&nbsp;[client-handshake.c:1445:client_setvolume_cbk]&nbsp;0-tcfstest-client-1:&nbsp;Server&nbsp;and&nbsp;Client&nbsp;lk-version&nbsp;numbers&nbsp;are&nbsp;not&nbsp;same,&nbsp;reopening&nbsp;the&nbsp;fds</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.937813]&nbsp;I&nbsp;[client-handshake.c:1636:select_server_supported_programs]&nbsp;0-tcfstest-client-5:&nbsp;Using&nbsp;Program&nbsp;GlusterFS&nbsp;3.3.0,&nbsp;Num&nbsp;(1298437),&nbsp;Version&nbsp;(330)</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.937918]&nbsp;I&nbsp;[client-handshake.c:453:client_set_lk_version_cbk]&nbsp;0-tcfstest-client-1:&nbsp;Server&nbsp;lk&nbsp;version&nbsp;=&nbsp;1</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.940899]&nbsp;I&nbsp;[client-handshake.c:1433:client_setvolume_cbk]&nbsp;0-tcfstest-client-5:&nbsp;Connected&nbsp;to&nbsp;125.210.140.20:24011,&nbsp;attached&nbsp;to&nbsp;remote&nbsp;volume&nbsp;'/mnt/p1/exp'.</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.940925]&nbsp;I&nbsp;[client-handshake.c:1445:client_setvolume_cbk]&nbsp;0-tcfstest-client-5:&nbsp;Server&nbsp;and&nbsp;Client&nbsp;lk-version&nbsp;numbers&nbsp;are&nbsp;not&nbsp;same,&nbsp;reopening&nbsp;the&nbsp;fds</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.940972]&nbsp;I&nbsp;[afr-common.c:3627:afr_notify]&nbsp;0-tcfstest-replicate-2:&nbsp;Subvolume&nbsp;'tcfstest-client-5'&nbsp;came&nbsp;back&nbsp;up;&nbsp;going&nbsp;online.</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.941097]&nbsp;I&nbsp;[client-handshake.c:453:client_set_lk_version_cbk]&nbsp;0-tcfstest-client-5:&nbsp;Server&nbsp;lk&nbsp;version&nbsp;=&nbsp;1</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.941334]&nbsp;I&nbsp;[client-handshake.c:1636:select_server_supported_programs]&nbsp;0-tcfstest-client-4:&nbsp;Using&nbsp;Program&nbsp;GlusterFS&nbsp;3.3.0,&nbsp;Num&nbsp;(1298437),&nbsp;Version&nbsp;(330)</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.944357]&nbsp;I&nbsp;[client-handshake.c:1433:client_setvolume_cbk]&nbsp;0-tcfstest-client-4:&nbsp;Connected&nbsp;to&nbsp;125.210.140.19:24011,&nbsp;attached&nbsp;to&nbsp;remote&nbsp;volume&nbsp;'/mnt/p1/exp'.</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.944381]&nbsp;I&nbsp;[client-handshake.c:1445:client_setvolume_cbk]&nbsp;0-tcfstest-client-4:&nbsp;Server&nbsp;and&nbsp;Client&nbsp;lk-version&nbsp;numbers&nbsp;are&nbsp;not&nbsp;same,&nbsp;reopening&nbsp;the&nbsp;fds</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.951156]&nbsp;I&nbsp;[client-handshake.c:453:client_set_lk_version_cbk]&nbsp;0-tcfstest-client-4:&nbsp;Server&nbsp;lk&nbsp;version&nbsp;=&nbsp;1</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.951332]&nbsp;I&nbsp;[afr-common.c:1964:afr_set_root_inode_on_first_lookup]&nbsp;0-tcfstest-replicate-0:&nbsp;added&nbsp;root&nbsp;inode</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.951889]&nbsp;I&nbsp;[afr-common.c:1964:afr_set_root_inode_on_first_lookup]&nbsp;0-tcfstest-replicate-1:&nbsp;added&nbsp;root&nbsp;inode</DIV>
<DIV>[2013-03-13&nbsp;09:53:52.952169]&nbsp;I&nbsp;[afr-common.c:1964:afr_set_root_inode_on_first_lookup]&nbsp;0-tcfstest-replicate-2:&nbsp;added&nbsp;root&nbsp;inode</DIV></DIV>
<HR style="WIDTH: 210px; HEIGHT: 1px" align=left color=#b5c4df SIZE=1>

<DIV><SPAN>Pippo</SPAN></DIV>
<DIV>&nbsp;</DIV>
<DIV 
style="BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0cm; PADDING-LEFT: 0cm; PADDING-RIGHT: 0cm; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<DIV 
style="PADDING-BOTTOM: 8px; PADDING-LEFT: 8px; PADDING-RIGHT: 8px; BACKGROUND: #efefef; COLOR: #000000; FONT-SIZE: 12px; PADDING-TOP: 8px">
<DIV><B>From:</B>&nbsp;<A href="mailto:pkarampu@redhat.com">Pranith Kumar 
K</A></DIV>
<DIV><B>Date:</B>&nbsp;2013-03-26&nbsp;01:31</DIV>
<DIV><B>To:</B>&nbsp;<A href="mailto:pippo0805@163.com">pippo0805</A></DIV>
<DIV><B>CC:</B>&nbsp;<A 
href="mailto:gluster-users@gluster.org">gluster-users</A></DIV>
<DIV><B>Subject:</B>&nbsp;Re: [Gluster-users] A problem when mount glusterfs via 
NFS</DIV></DIV></DIV>
<DIV>
<DIV class=FoxDiv20130326074000788998>
<DIV class=moz-cite-prefix>On 03/25/2013 08:23 AM, Pippo wrote:<BR></DIV>
<BLOCKQUOTE cite=mid:201303251053374967568@163.com type="cite">
  <STYLE>BLOCKQUOTE {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px; MARGIN-LEFT: 2em
}
OL {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
UL {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
P {
        MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
</STYLE>

  <META name=GENERATOR content="MSHTML 8.00.7601.17514">
  <DIV>HI:</DIV>
  <DIV>&nbsp;</DIV>
  <DIV style="TEXT-INDENT: 2em">
  <DIV>I&nbsp;run&nbsp;glusterfs&nbsp;with&nbsp;four&nbsp;nodes,&nbsp;2x2&nbsp;Distributed-Replicate.</DIV>
  <DIV>I&nbsp;mounted&nbsp;it&nbsp;via&nbsp;fuse&nbsp;and&nbsp;did&nbsp;some&nbsp;test,&nbsp;it&nbsp;was&nbsp;ok.</DIV>
  <DIV>However&nbsp;when&nbsp;I&nbsp;mounted&nbsp;it&nbsp;via&nbsp;nfs,&nbsp;a&nbsp;problem&nbsp;was&nbsp;found:</DIV>
  <DIV>&nbsp;&nbsp;&nbsp;&nbsp;</DIV>
  <DIV style="TEXT-INDENT: 2em">&nbsp;&nbsp;&nbsp; 
  When&nbsp;I&nbsp;copied&nbsp;200G&nbsp;files&nbsp;to&nbsp;the&nbsp;glusterfs,&nbsp;the&nbsp;glusterfs&nbsp;process&nbsp;in&nbsp;the&nbsp;server&nbsp;node(mounted&nbsp;by&nbsp;client)&nbsp;was&nbsp;killed&nbsp;because&nbsp;of&nbsp;OOM,&nbsp;</DIV>
  <DIV>and&nbsp;all&nbsp;terminals&nbsp;of&nbsp;the&nbsp;client&nbsp;were&nbsp;hung.&nbsp;Trying&nbsp;to&nbsp;test&nbsp;for&nbsp;many&nbsp;times,&nbsp;I&nbsp;got&nbsp;the&nbsp;same&nbsp;result.&nbsp;The&nbsp;heavier&nbsp;load&nbsp;I&nbsp;pushed&nbsp;via&nbsp;the</DIV>
  <DIV>client,&nbsp;the&nbsp;faster&nbsp;glusterfs&nbsp;process&nbsp;been&nbsp;killed.&nbsp;I&nbsp;run&nbsp;"top"&nbsp;in&nbsp;the&nbsp;server,&nbsp;found&nbsp;that&nbsp;the&nbsp;glusterfs&nbsp;process&nbsp;eat&nbsp;MEM&nbsp;very&nbsp;fast,&nbsp; 
  and&nbsp;</DIV>
  <DIV>never&nbsp;gone&nbsp;down&nbsp;until&nbsp;it&nbsp;was&nbsp;killed.&nbsp;I&nbsp;think&nbsp;it&nbsp;is&nbsp;a&nbsp;bug&nbsp;of&nbsp;glusterfs&nbsp;process,&nbsp;it&nbsp;leak&nbsp;memory.</DIV>
  <DIV>&nbsp;&nbsp;&nbsp;</DIV>
  <DIV>I&nbsp;google&nbsp;"glusterfs&nbsp;OOM"&nbsp;but&nbsp;can&nbsp;not&nbsp;find&nbsp;any&nbsp;solutions.&nbsp;Is&nbsp;anyone&nbsp;know 
  about&nbsp;this&nbsp;problem&nbsp;and&nbsp;give&nbsp;me&nbsp;some&nbsp;tips?&nbsp;Many&nbsp;thanks!</DIV></DIV>
  <DIV>&nbsp;</DIV>
  <HR style="WIDTH: 210px; HEIGHT: 1px" align=left color=#b5c4df SIZE=1>

  <DIV><SPAN>Pippo</SPAN></DIV><BR>
  <FIELDSET class=mimeAttachmentHeader></FIELDSET> <BR><PRE wrap="">_______________________________________________
Gluster-users mailing list
<A class=moz-txt-link-abbreviated href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</A>
<A class=moz-txt-link-freetext href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</A></PRE></BLOCKQUOTE>hi,<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Could you let us know the version of glusterfs you were using? nfs server Logs 
of that run would help us if you could attach that to this 
mail.<BR><BR>Pranith<BR></DIV></DIV></BODY></HTML>