<div dir="ltr">Hello all.<div><br></div><div>I have a very curious puzzle here, an maybe you want to chip in your opinion.</div><div><br></div><div>My current setup, which I was hoping would be the last, is the following:</div>
<div><br></div><div>One node: 1xDell 2950 8 GB ram,with an external RAID (Ataboy2) connected via a SCSI card, issuing about 70 MB/s (slow, I know). </div><div><br></div><div>One node Dell 2950 with internal raid. Raid 5.</div>
<div><br></div><div>both Dells have 4 (yep, four) bonded NICs using the useless mode 6. I recently learned that REAL link aggregation is switch dependent, so, if you want SPEED, never mind playing around with software.</div>
<div><br></div><div>I have a REPLICATED volume using both servers with the respective bricks.</div><div><br></div><div>As part of the scenario I have an Isilon, which is my primary storage for a few (10) fairly gig VMware images. <br>
</div><div><br></div><div>One of the nodes mounts a NFS share to the isilon and mounts the gluster volume using the native glusterfs option.</div><div><br></div><div>I am on a unique situation where I can afford suspending the VM servers for a few hours for a backup, so I wrote a nice simple bash script, ran from that node, that does exactly this:</div>
<div><br></div><div>pauses the VM</div><div>uses cp to copy the pertinent files from the Isilon to the gluster volume.</div><div>resumes the VM</div><div><br></div><div>and repeats that for all of the servers.</div><div><br>
</div><div>Simple and elegant, I like to think.</div><div><br></div><div>BUT, here is the trouble:</div><div><br></div><div>For 20 SOLID minutes the system sits, reporting a steady connection to the isilon, receiving at 740 Mbps. Impressive you would thing, right ? That would be something like 150 GB of data.</div>
<div><br></div><div>Well, the problem is, NOTHING is written to the disks. NO disk at all. And, with 8 GM of ram, it is not going to the memory either.</div><div><br></div><div>There is NOTHING on gluster-related logs. Actually, the only entries there are from two days ago, when I last rebooted the system.</div>
<div><br></div><div>There is nothing on system logs either, and, regarding network communication, there is no data flow to ANY other peer or point on the servers. </div><div><br></div><div>After those 20 minutes of going fast nowhere, THEN the system decides to start transmitting data to the other node, bandwidth usage falls to around 270-300 Mbps, now FINALLY we have data recorded to both volumes.</div>
<div><br></div><div>While the issue is happening, gluster-related processes are doing nothing. No processing reported by top.</div><div><br></div><div>Any idea about what is going on here ?</div><div><br></div><div>I am not even sure this is gluster, but, well, I can't think of anything else right now.</div>
<div><br></div><div>on the network, x.92 is the isilon.</div><div>x.23 and x.24 are the nodes.</div><div><br></div><div>While that bizarre behavior was taking place, a quick tcpdump showed:</div><div><br></div><div><br></div>
<div><div>19:31:41.355065 IP 10.0.1.24.ssh > 10.0.1.23.50250: Flags [P.], seq 20534432:20534816, ack 7393, win 108, options [nop,nop,TS val 209602327 ecr 209546443], length 384</div><div>19:31:41.355123 IP 10.0.1.24.ssh > 10.0.1.23.50250: Flags [P.], seq 20534816:20535648, ack 7393, win 108, options [nop,nop,TS val 209602327 ecr 209546443], length 832</div>
<div>19:31:41.355164 IP 10.0.1.23.50250 > 10.0.1.24.ssh: Flags [.], ack 20534432, win 1122, options [nop,nop,TS val 209546443 ecr 209602327], length 0</div><div>19:31:41.355173 IP 10.0.1.24.ssh > 10.0.1.23.50250: Flags [P.], seq 20535648:20535856, ack 7393, win 108, options [nop,nop,TS val 209602327 ecr 209546443], length 208</div>
<div>19:31:41.355188 IP 10.0.1.92.nfs > 10.0.1.24.954: Flags [.], seq 602055725:602064673, ack 1211761, win 65535, options [nop,nop,TS val 200506019 ecr 209602324], length 8948</div><div>19:31:41.355208 IP 10.0.1.24.954 > 10.0.1.92.nfs: Flags [.], ack 602064673, win 6530, options [nop,nop,TS val 209602327 ecr 200506019], length 0</div>
<div>19:31:41.355213 IP 10.0.1.92.nfs > 10.0.1.24.954: Flags [.], seq 602064673:602073621, ack 1211761, win 65535, options [nop,nop,TS val 200506019 ecr 209602324], length 8948</div><div>19:31:41.355218 IP 10.0.1.24.ssh > 10.0.1.23.50250: Flags [P.], seq 20535856:20536064, ack 7393, win 108, options [nop,nop,TS val 209602327 ecr 209546443], length 208</div>
<div>19:31:41.355224 IP 10.0.1.92.nfs > 10.0.1.24.954: Flags [.], seq 602073621:602082569, ack 1211761, win 65535, options [nop,nop,TS val 200506019 ecr 209602324], length 8948</div><div>19:31:41.355235 IP 10.0.1.24.954 > 10.0.1.92.nfs: Flags [.], ack 602082569, win 6530, options [nop,nop,TS val 209602327 ecr 200506019], length 0</div>
<div>19:31:41.355239 IP 10.0.1.24.ssh > 10.0.1.23.50250: Flags [P.], seq 20536064:20536416, ack 7393, win 108, options [nop,nop,TS val 209602327 ecr 209546443], length 352</div><div>19:31:41.355266 IP 10.0.1.23.50250 > 10.0.1.24.ssh: Flags [.], ack 20535648, win 1122, options [nop,nop,TS val 209546443 ecr 209602327], length 0</div>
<div>19:31:41.355360 IP 10.0.1.23.50250 > 10.0.1.24.ssh: Flags [.], ack 20536416, win 1122, options [nop,nop,TS val 209546443 ecr 209602327], length 0</div><div>19:31:41.355429 IP 10.0.1.92.nfs > 10.0.1.24.954: Flags [.], seq 602082569:602091517, ack 1211761, win 65535, options [nop,nop,TS val 200506019 ecr 209602324], length 8948</div>
</div></div>