<div dir="ltr">Hello there,<div><br></div><div>I've 6 servers creating a 50TB storage. My setup is Distribution + Replication. My servers are named: storage01 - storage06, and they are running Ubuntu 12.04 64-bit.</div>
<div>Today I decided to install available packages updates on the servers. So I started with storage06. After the installation of new packages and a new kernel, I decided to reboot it to use the newly installed kernel. I issued the reboot and withing minutes, the server was back again. But gluster on it seems to be offline:</div>
<div><br></div><div>When I run the below command on storage01 (The server where I created all the volumes) I get the below:</div><div><br></div><div><div>root@storage01:~# gluster peer status</div><div>Number of Peers: 5</div>
<div><br></div><div>Hostname: storage05</div><div>Uuid: 9274b525-8089-4cc7-afd8-5031b7e2774e</div><div>State: Peer in Cluster (Connected)</div><div><br></div><div>Hostname: storage02</div><div>Uuid: a0d68c85-b34a-4052-9d45-cbdd9b08ad18</div>
<div>State: Peer in Cluster (Connected)</div><div><br></div><div>Hostname: storage06</div><div>Uuid: 5aadb3d0-d7bc-447f-83a7-368104b4ddc4</div><div>State: Peer in Cluster (Connected)</div><div><br></div><div>Hostname: storage03</div>
<div>Uuid: 94d0375a-31f2-4bb1-9a5d-647c20eedbcf</div><div>State: Peer in Cluster (Connected)</div><div><br></div><div>Hostname: storage04</div><div>Uuid: 888cb603-ffbf-4f09-a0a4-890170519eb9</div><div>State: Peer in Cluster (Connected)</div>
</div><div><div><br></div><div><br></div><div>But when I run volume info, I get:</div><div><br></div><div><div>gluster volume status all detail</div><div>Status of volume: orig</div><div>------------------------------------------------------------------------------</div>
<div>Brick : Brick storage01:/mnt/storage/orig</div><div>Port : 24012 </div><div>Online : Y </div><div>Pid : 775 </div>
<div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div><div>Inode Size : 256 </div>
<div>Disk Space Free : 11.5TB </div><div>Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>Free Inodes : 3864568149 </div><div>
------------------------------------------------------------------------------</div><div>Brick : Brick storage02:/mnt/storage/orig</div><div>Port : 24012 </div><div>Online : Y </div>
<div>Pid : 22838 </div><div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div>
<div>Inode Size : 256 </div><div>Disk Space Free : 8.3TB </div><div>Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>
Free Inodes : 3864445198 </div><div>------------------------------------------------------------------------------</div><div>Brick : Brick storage03:/mnt/storage/orig</div><div>Port : 24012 </div>
<div>Online : Y </div><div>Pid : 11342 </div><div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div>
<div>Mount Options : rw </div><div>Inode Size : 256 </div><div>Disk Space Free : 11.4TB </div><div>Total Disk Space : 18.0TB </div><div>
Inode Count : 3865470528 </div><div>Free Inodes : 3864561821 </div><div>------------------------------------------------------------------------------</div><div>Brick : Brick storage04:/mnt/storage/orig</div>
<div>Port : 24012 </div><div>Online : Y </div><div>Pid : 10887 </div><div>File System : xfs </div><div>
Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div><div>Inode Size : 256 </div><div>Disk Space Free : 11.3TB </div><div>
Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>Free Inodes : 3864557582 </div><div>------------------------------------------------------------------------------</div>
<div>Brick : Brick storage05:/mnt/storage/orig</div><div>Port : 24012 </div><div>Online : Y </div><div>Pid : 32098 </div>
<div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div><div>Inode Size : 256 </div>
<div>Disk Space Free : 11.4TB </div><div>Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>Free Inodes : 3864561308 </div><div>
------------------------------------------------------------------------------</div><div>Brick : Brick storage06:/mnt/storage/orig</div><div>Port : 24012 </div><div>Online : N </div>
<div>Pid : N/A </div><div>File System : N/A </div><div>Device : N/A </div><div>Mount Options : N/A </div><div>
Inode Size : N/A </div><div>Disk Space Free : 0Bytes </div><div>Total Disk Space : 0Bytes </div><div>Inode Count : N/A </div><div>Free Inodes : N/A </div>
<div> </div><div>Status of volume: mp3</div><div>------------------------------------------------------------------------------</div><div>Brick : Brick storage01:/mnt/storage/mp3</div><div>Port : 24009 </div>
<div>Online : Y </div><div>Pid : 737 </div><div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div>
<div>Mount Options : rw </div><div>Inode Size : 256 </div><div>Disk Space Free : 11.5TB </div><div>Total Disk Space : 18.0TB </div><div>
Inode Count : 3865470528 </div><div>Free Inodes : 3864568149 </div><div>------------------------------------------------------------------------------</div><div>Brick : Brick storage02:/mnt/storage/mp3</div>
<div>Port : 24009 </div><div>Online : Y </div><div>Pid : 22802 </div><div>File System : xfs </div><div>
Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div><div>Inode Size : 256 </div><div>Disk Space Free : 8.3TB </div><div>
Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>Free Inodes : 3864445198 </div><div>------------------------------------------------------------------------------</div>
<div>Brick : Brick storage03:/mnt/storage/mp3</div><div>Port : 24009 </div><div>Online : Y </div><div>Pid : 11306 </div>
<div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div><div>Inode Size : 256 </div>
<div>Disk Space Free : 11.4TB </div><div>Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>Free Inodes : 3864561821 </div><div>
------------------------------------------------------------------------------</div><div>Brick : Brick storage04:/mnt/storage/mp3</div><div>Port : 24009 </div><div>Online : Y </div>
<div>Pid : 10853 </div><div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div>
<div>Inode Size : 256 </div><div>Disk Space Free : 11.3TB </div><div>Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>
Free Inodes : 3864557582 </div><div>------------------------------------------------------------------------------</div><div>Brick : Brick storage05:/mnt/storage/mp3</div><div>Port : 24009 </div>
<div>Online : Y </div><div>Pid : 32062 </div><div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div>
<div>Mount Options : rw </div><div>Inode Size : 256 </div><div>Disk Space Free : 11.4TB </div><div>Total Disk Space : 18.0TB </div><div>
Inode Count : 3865470528 </div><div>Free Inodes : 3864561308 </div><div>------------------------------------------------------------------------------</div><div>Brick : Brick storage06:/mnt/storage/mp3</div>
<div>Port : 24009 </div><div>Online : N </div><div>Pid : N/A </div><div>File System : N/A </div><div>
Device : N/A </div><div>Mount Options : N/A </div><div>Inode Size : N/A </div><div>Disk Space Free : 0Bytes </div><div>Total Disk Space : 0Bytes </div>
<div>Inode Count : N/A </div><div>Free Inodes : N/A </div><div> </div><div>Status of volume: flv</div><div>------------------------------------------------------------------------------</div>
<div>Brick : Brick storage01:/mnt/storage/flv</div><div>Port : 24011 </div><div>Online : Y </div><div>Pid : 435 </div>
<div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div><div>Inode Size : 256 </div>
<div>Disk Space Free : 11.5TB </div><div>Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>Free Inodes : 3864568149 </div><div>
------------------------------------------------------------------------------</div><div>Brick : Brick storage02:/mnt/storage/flv</div><div>Port : 24011 </div><div>Online : Y </div>
<div>Pid : 22544 </div><div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div>
<div>Inode Size : 256 </div><div>Disk Space Free : 8.3TB </div><div>Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>
Free Inodes : 3864445198 </div><div>------------------------------------------------------------------------------</div><div>Brick : Brick storage03:/mnt/storage/flv</div><div>Port : 24011 </div>
<div>Online : Y </div><div>Pid : 11051 </div><div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div>
<div>Mount Options : rw </div><div>Inode Size : 256 </div><div>Disk Space Free : 11.4TB </div><div>Total Disk Space : 18.0TB </div><div>
Inode Count : 3865470528 </div><div>Free Inodes : 3864561821 </div><div>------------------------------------------------------------------------------</div><div>Brick : Brick storage04:/mnt/storage/flv</div>
<div>Port : 24011 </div><div>Online : Y </div><div>Pid : 10595 </div><div>File System : xfs </div><div>
Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div><div>Inode Size : 256 </div><div>Disk Space Free : 11.3TB </div><div>
Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>Free Inodes : 3864557582 </div><div>------------------------------------------------------------------------------</div>
<div>Brick : Brick storage05:/mnt/storage/flv</div><div>Port : 24011 </div><div>Online : Y </div><div>Pid : 31798 </div>
<div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div><div>Inode Size : 256 </div>
<div>Disk Space Free : 11.4TB </div><div>Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>Free Inodes : 3864561308 </div><div>
------------------------------------------------------------------------------</div><div>Brick : Brick storage06:/mnt/storage/flv</div><div>Port : 24011 </div><div>Online : N </div>
<div>Pid : N/A </div><div>File System : N/A </div><div>Device : N/A </div><div>Mount Options : N/A </div><div>
Inode Size : N/A </div><div>Disk Space Free : 0Bytes </div><div>Total Disk Space : 0Bytes </div><div>Inode Count : N/A </div><div>Free Inodes : N/A </div>
<div> </div><div>Status of volume: mp4</div><div>------------------------------------------------------------------------------</div><div>Brick : Brick storage01:/mnt/storage/mp4</div><div>Port : 24010 </div>
<div>Online : Y </div><div>Pid : 698 </div><div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div>
<div>Mount Options : rw </div><div>Inode Size : 256 </div><div>Disk Space Free : 11.5TB </div><div>Total Disk Space : 18.0TB </div><div>
Inode Count : 3865470528 </div><div>Free Inodes : 3864568149 </div><div>------------------------------------------------------------------------------</div><div>Brick : Brick storage02:/mnt/storage/mp4</div>
<div>Port : 24010 </div><div>Online : Y </div><div>Pid : 22756 </div><div>File System : xfs </div><div>
Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div><div>Inode Size : 256 </div><div>Disk Space Free : 8.3TB </div><div>
Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>Free Inodes : 3864445198 </div><div>------------------------------------------------------------------------------</div>
<div>Brick : Brick storage03:/mnt/storage/mp4</div><div>Port : 24010 </div><div>Online : Y </div><div>Pid : 11272 </div>
<div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div><div>Inode Size : 256 </div>
<div>Disk Space Free : 11.4TB </div><div>Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>Free Inodes : 3864561821 </div><div>
------------------------------------------------------------------------------</div><div>Brick : Brick storage04:/mnt/storage/mp4</div><div>Port : 24010 </div><div>Online : Y </div>
<div>Pid : 10818 </div><div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div><div>Mount Options : rw </div>
<div>Inode Size : 256 </div><div>Disk Space Free : 11.3TB </div><div>Total Disk Space : 18.0TB </div><div>Inode Count : 3865470528 </div><div>
Free Inodes : 3864557582 </div><div>------------------------------------------------------------------------------</div><div>Brick : Brick storage05:/mnt/storage/mp4</div><div>Port : 24010 </div>
<div>Online : Y </div><div>Pid : 32027 </div><div>File System : xfs </div><div>Device : /dev/mapper/storage_vg-videos</div>
<div>Mount Options : rw </div><div>Inode Size : 256 </div><div>Disk Space Free : 11.4TB </div><div>Total Disk Space : 18.0TB </div><div>
Inode Count : 3865470528 </div><div>Free Inodes : 3864561308 </div><div>------------------------------------------------------------------------------</div><div>Brick : Brick storage06:/mnt/storage/mp4</div>
<div>Port : 24010 </div><div>Online : N </div><div>Pid : N/A </div><div>File System : N/A </div><div>
Device : N/A </div><div>Mount Options : N/A </div><div>Inode Size : N/A </div><div>Disk Space Free : 0Bytes </div><div>Total Disk Space : 0Bytes </div>
<div>Inode Count : N/A </div><div>Free Inodes : N/A </div></div><div><br></div><div><br></div><div>As you can see, all bricks from storage06 are offiline. I tried to restart gluster daemon many times, I rebooted, and I even stopped all volumes and started them again but the same result. My main issue is that this is a production environment, and the whole load is going to storage05 (load 35 now and all CPUs 100% full) and I can't do a lot of trials on it.</div>
<div><br></div><div>Your advice please?</div><div><br></div><div>Thanks & Regards,</div><div><br></div>
</div></div>