Hello,<br><br>We use gluster in our production environment and recently encountered an unrecoverable error which was solved only by deleting the existing volume and local stored files and recreating from scratch.<br><br>I am now playing with a test environment which almost mirrors the prod and I can always reproduce the problem.<br>
We have websites on two servers which use gluster for common usage files. We also use DNS round robin for request balancing (this is a main element in the scenario).<br><br>Setup: Two servers running Gentoo 2.0.3 kernel 3.0.6, glusterfs 3.2.5<br>
Gluster commands:<br>gluster volume create vol-replication replica 2 transport tcp 10.0.2.14:/local 10.0.2.15:/local<br>gluster volume start vol-replication<br>gluster volume set vol-replication network.ping-timeout 1<br>
node1 (10.0.2.14): mount -t glusterfs 10.0.2.14:/vol-replication /a<br>node2 (10.0.2.15): mount -t glusterfs 10.0.2.15:/vol-replication /a<br><br>Now assume that connectivity between the two nodes has failed, but they can still be accessed from the outside world and files can be written on them through Apache.<br>
Request 1 -> 10.0.2.14 -> creates file howareyou<br>Request 2 -> 10.0.2.15 -> creates file hello<br>At some point, connectivity between the two nodes recovers and disaster strikes: <br>ls /a<br>ls: cannot access /a: Input/output error<br>
<br>Simulation follows:<br>step 1<br>node1:<br>iptables -I INPUT 1 -s 10.0.2.15 -j DROP (connectivity loss simulation)<br>touch /a/howareyou<br><br>node2:<br>touch /a/hello<br><br>step 2<br>node1:<br>iptables -D INPUT 1 (connectivity recovery)<br>
ls /a<br>ls: cannot access /a: Input/output error<br><br>node2:<br>ls /a<br>ls: cannot access /a: Input/output error<br><br>The only way to recover this was to delete the offending files. This was easy to do on the test environment because there were two files involved, but on the prod environment we had many more and I managed to recover only after deleting the gluster volume and the local content including the local storage directory itself! Nothing else of what I tried (stopping volume, recreating volume, emptying the local storage directory, remounting, restarting gluster) worked.<br>
<br>Any hint on how one could recover from this sort of situation?<br>Thank you.<br>