<div dir="ltr">forgot to add this output:<div><br></div><div><div>root@stor1:~# du -sh /exports/fast-test/150G/images/</div><div>2.8G    /exports/fast-test/150G/images/</div></div><div><br></div><div><div>root@stor2:~# du -sh /exports/fast-test/150G/images/</div>
<div>2.9G    /exports/fast-test/150G/images/</div></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-08-05 11:25 GMT+03:00 Pranith Kumar Karampuri <span dir="ltr">&lt;<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>&gt;</span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF"><div class="">
    <br>
    <div>On 08/05/2014 01:10 PM, Roman wrote:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr">Ahha! For some reason I was not able to start the
        VM anymore, Proxmox VE told me, that it is not able to read the
        qcow2 header due to permission is denied for some reason. So I
        just deleted that file and created a new VM. And the nex message
        I&#39;ve got was this:</div>
    </blockquote></div>
    Seems like these are the messages where you took down the bricks
    before self-heal. Could you restart the run waiting for self-heals
    to complete before taking down the next brick?<span class="HOEnZb"><font color="#888888"><br>
    <br>
    Pranith</font></span><div><div class="h5"><br>
    <blockquote type="cite">
      <div dir="ltr">
        <div>
          <br>
          <div><br>
          </div>
          <div>
            <div>[2014-08-05 07:31:25.663412] E
              [afr-self-heal-common.c:197:afr_sh_print_split_brain_log]
              0-HA-fast-150G-PVE1-replicate-0: Unable to self-heal
              contents of &#39;/images/124/vm-124-disk-1.qcow2&#39; (possible
              split-brain). Please delete the file from all but the
              preferred subvolume.- Pending matrix:  [ [ 0 60 ] [ 11 0 ]
              ]</div>
            <div>[2014-08-05 07:31:25.663955] E
              [afr-self-heal-common.c:2262:afr_self_heal_completion_cbk]
              0-HA-fast-150G-PVE1-replicate-0: background  data
              self-heal failed on /images/124/vm-124-disk-1.qcow2</div>
          </div>
          <div><br>
          </div>
        </div>
      </div>
      <div class="gmail_extra"><br>
        <br>
        <div class="gmail_quote">2014-08-05 10:13 GMT+03:00 Pranith
          Kumar Karampuri <span dir="ltr">&lt;<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>&gt;</span>:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div text="#000000" bgcolor="#FFFFFF"> I just responded to
              your earlier mail about how the log looks. The log comes
              on the mount&#39;s logfile<span><font color="#888888"><br>
                  <br>
                  Pranith</font></span>
              <div>
                <div><br>
                  <div>On 08/05/2014 12:41 PM, Roman wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <div dir="ltr">Ok, so I&#39;ve waited enough, I think.
                      Had no any traffic on switch ports between
                      servers. Could not find any suitable log message
                      about completed self-heal (waited about 30
                      minutes). Plugged out the other server&#39;s UTP cable
                      this time and got in the same situation:
                      <div>
                        <div>root@gluster-test1:~# cat /var/log/dmesg</div>
                        <div>-bash: /bin/cat: Input/output error</div>
                      </div>
                      <div><br>
                      </div>
                      <div>brick logs:</div>
                      <div>
                        <div>[2014-08-05 07:09:03.005474] I
                          [server.c:762:server_rpc_notify]
                          0-HA-fast-150G-PVE1-server: disconnecting
                          connectionfrom
                          pve1-27649-2014/08/04-13:27:54:720789-HA-fast-150G-PVE1-client-0-0</div>
                        <div>[2014-08-05 07:09:03.005530] I
                          [server-helpers.c:729:server_connection_put]
                          0-HA-fast-150G-PVE1-server: Shutting down
                          connection
                          pve1-27649-2014/08/04-13:27:54:720789-HA-fast-150G-PVE1-client-0-0</div>
                        <div>[2014-08-05 07:09:03.005560] I
                          [server-helpers.c:463:do_fd_cleanup]
                          0-HA-fast-150G-PVE1-server: fd cleanup on
                          /images/124/vm-124-disk-1.qcow2</div>
                        <div>[2014-08-05 07:09:03.005797] I
                          [server-helpers.c:617:server_connection_destroy]
                          0-HA-fast-150G-PVE1-server: destroyed
                          connection of
                          pve1-27649-2014/08/04-13:27:54:720789-HA-fast-150G-PVE1-client-0-0</div>
                      </div>
                      <div><br>
                      </div>
                      <div><br>
                      </div>
                      <div><br>
                      </div>
                    </div>
                    <div class="gmail_extra"><br>
                      <br>
                      <div class="gmail_quote">2014-08-05 9:53 GMT+03:00
                        Pranith Kumar Karampuri <span dir="ltr">&lt;<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>&gt;</span>:<br>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                          <div text="#000000" bgcolor="#FFFFFF"> Do you
                            think it is possible for you to do these
                            tests on the latest version 3.5.2? &#39;gluster
                            volume heal &lt;volname&gt; info&#39; would give
                            you that information in versions &gt; 3.5.1.<br>
                            Otherwise you will have to check it from
                            either the logs, there will be self-heal
                            completed message on the mount logs (or) by
                            observing &#39;getfattr -d -m. -e hex
                            &lt;image-file-on-bricks&gt;&#39;<span><font color="#888888"><br>
                                <br>
                                Pranith</font></span>
                            <div>
                              <div><br>
                                <br>
                                <div>On 08/05/2014 12:09 PM, Roman
                                  wrote:<br>
                                </div>
                                <blockquote type="cite">
                                  <div dir="ltr">Ok, I understand. I
                                    will try this shortly.
                                    <div>How can I be sure, that healing
                                      process is done, if I am not able
                                      to see its status?</div>
                                  </div>
                                  <div class="gmail_extra"><br>
                                    <br>
                                    <div class="gmail_quote">2014-08-05
                                      9:30 GMT+03:00 Pranith Kumar
                                      Karampuri <span dir="ltr">&lt;<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>&gt;</span>:<br>
                                      <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                        <div text="#000000" bgcolor="#FFFFFF"> Mounts will
                                          do the healing, not the
                                          self-heal-daemon. The problem
                                          I feel is that whichever
                                          process does the healing has
                                          the latest information about
                                          the good bricks in this
                                          usecase. Since for VM usecase,
                                          mounts should have the latest
                                          information, we should let the
                                          mounts do the healing. If the
                                          mount accesses the VM image
                                          either by someone doing
                                          operations inside the VM or
                                          explicit stat on the file it
                                          should do the healing.<span><font color="#888888"><br>
                                              <br>
                                              Pranith.</font></span>
                                          <div>
                                            <div><br>
                                              <br>
                                              <div>On 08/05/2014 10:39
                                                AM, Roman wrote:<br>
                                              </div>
                                              <blockquote type="cite">
                                                <div dir="ltr">Hmmm, you
                                                  told me to turn it
                                                  off. Did I understood
                                                  something wrong? After
                                                  I issued the command
                                                  you&#39;ve sent me, I was
                                                  not able to watch the
                                                  healing process, it
                                                  said, it won&#39;t be
                                                  healed, becouse its
                                                  turned off.</div>
                                                <div class="gmail_extra"><br>
                                                  <br>
                                                  <div class="gmail_quote">2014-08-05

                                                    5:39 GMT+03:00
                                                    Pranith Kumar
                                                    Karampuri <span dir="ltr">&lt;<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>&gt;</span>:<br>
                                                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                      <div text="#000000" bgcolor="#FFFFFF">
                                                        You didn&#39;t
                                                        mention anything
                                                        about
                                                        self-healing.
                                                        Did you wait
                                                        until the
                                                        self-heal is
                                                        complete?<span><font color="#888888"><br>
                                                          <br>
                                                          Pranith</font></span>
                                                        <div>
                                                          <div><br>
                                                          <div>On
                                                          08/04/2014
                                                          05:49 PM,
                                                          Roman wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div>Hi!</div>
                                                          <div>Result is
                                                          pretty same. I
                                                          set the switch
                                                          port down for
                                                          1st server, it
                                                          was ok. Then
                                                          set it up back
                                                          and set other
                                                          server&#39;s port
                                                          off. and it
                                                          triggered IO
                                                          error on two
                                                          virtual
                                                          machines: one
                                                          with local
                                                          root FS but
                                                          network
                                                          mounted
                                                          storage. and
                                                          other with
                                                          network root
                                                          FS. 1st gave
                                                          an error on
                                                          copying to or
                                                          from the
                                                          mounted
                                                          network disk,
                                                          other just
                                                          gave me an
                                                          error for even
                                                          reading
                                                          log.files.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>
                                                          <div>cat:
                                                          /var/log/alternatives.log:
                                                          Input/output
                                                          error<br>
                                                          </div>
                                                          <div>then I
                                                          reset the kvm
                                                          VM and it said
                                                          me, there is
                                                          no boot
                                                          device. Next I
                                                          virtually
                                                          powered it off
                                                          and then back
                                                          on and it has
                                                          booted.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>By the
                                                          way, did I
                                                          have to
                                                          start/stop
                                                          volume?</div>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>&gt;&gt; <span style="font-family:arial,sans-serif;font-size:13px">Could you do the
                                                          following and
                                                          test it again?</span></div>
                                                          <span style="font-family:arial,sans-serif;font-size:13px">&gt;&gt;




                                                          gluster volume
                                                          set
                                                          &lt;volname&gt;
                                                          cluster.self-heal-daemon

                                                          off</span><br style="font-family:arial,sans-serif;font-size:13px">
                                                          <br style="font-family:arial,sans-serif;font-size:13px">
                                                          <span style="font-family:arial,sans-serif;font-size:13px">&gt;&gt;Pranith</span>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          <div class="gmail_extra"><br>
                                                          <br>
                                                          <div class="gmail_quote">2014-08-04


                                                          14:10
                                                          GMT+03:00
                                                          Pranith Kumar
                                                          Karampuri <span dir="ltr">&lt;<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>&gt;</span>:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div text="#000000" bgcolor="#FFFFFF">
                                                          <div> <br>
                                                          <div>On
                                                          08/04/2014
                                                          03:33 PM,
                                                          Roman wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div><span style="font-family:arial,sans-serif;font-size:13px">Hello!</span></div>
                                                          <div><span style="font-family:arial,sans-serif;font-size:13px"><br>
                                                          </span></div>
                                                          <span style="font-family:arial,sans-serif;font-size:13px">Facing





                                                          the same
                                                          problem as
                                                          mentioned
                                                          here:</span>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">
                                                          <br>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px"><a href="http://supercolony.gluster.org/pipermail/gluster-users/2014-April/039959.html" target="_blank">http://supercolony.gluster.org/pipermail/gluster-users/2014-April/039959.html</a></div>

                                                          <div style="font-family:arial,sans-serif;font-size:13px"><br>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">my




                                                          set up is up
                                                          and running,
                                                          so i&#39;m ready
                                                          to help you
                                                          back with
                                                          feedback.</div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">
                                                          <br>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">setup:</div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">proxmox





                                                          server as
                                                          client</div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">
                                                          2 gluster
                                                          physical
                                                           servers</div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px"><br>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">server





                                                          side and
                                                          client side
                                                          both running
                                                          atm 3.4.4
                                                          glusterfs from
                                                          gluster repo.</div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px"><br>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">the





                                                          problem is:</div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px"><br>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">
                                                          1. craeted
                                                          replica
                                                          bricks.</div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">2.





                                                          mounted in
                                                          proxmox (tried
                                                          both promox
                                                          ways: via GUI
                                                          and fstab
                                                          (with backup
                                                          volume line),
                                                          btw while
                                                          mounting via
                                                          fstab I&#39;m
                                                          unable to
                                                          launch a VM
                                                          without cache,
                                                          meanwhile
                                                          direct-io-mode
                                                          is enabled in
                                                          fstab line)</div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">3.





                                                          installed VM</div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">4.





                                                          bring one
                                                          volume down -
                                                          ok</div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">
                                                          5. bringing
                                                          up, waiting
                                                          for sync is
                                                          done.</div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">6.





                                                          bring other
                                                          volume down -
                                                          getting IO
                                                          errors on VM
                                                          guest and not
                                                          able to
                                                          restore the VM
                                                          after I reset
                                                          the VM via
                                                          host. It says
                                                          (no bootable
                                                          media). After
                                                          I shut it down
                                                          (forced) and
                                                          bring back up,
                                                          it boots.</div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          Could you do
                                                          the following
                                                          and test it
                                                          again?<br>
                                                          gluster volume
                                                          set
                                                          &lt;volname&gt;
                                                          cluster.self-heal-daemon

                                                          off<br>
                                                          <br>
                                                          Pranith<br>
                                                          <blockquote type="cite">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div style="font-family:arial,sans-serif;font-size:13px"><br>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">Need





                                                          help. Tried
                                                          3.4.3, 3.4.4.</div>
                                                          <div style="font-family:arial,sans-serif;font-size:13px">Still





                                                          missing pkg-s
                                                          for 3.4.5 for
                                                          debian and
                                                          3.5.2 (3.5.1
                                                          always gives a
                                                          healing error
                                                          for some
                                                          reason)</div>
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          Best regards,<br>
                                                          Roman. </div>
                                                          <br>
                                                          <fieldset></fieldset>
                                                          <br>
                                                          </div>
                                                          <pre>_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          <br clear="all">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          Best regards,<br>
                                                          Roman. </div>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </blockquote>
                                                  </div>
                                                  <br>
                                                  <br clear="all">
                                                  <div><br>
                                                  </div>
                                                  -- <br>
                                                  Best regards,<br>
                                                  Roman. </div>
                                              </blockquote>
                                              <br>
                                            </div>
                                          </div>
                                        </div>
                                      </blockquote>
                                    </div>
                                    <br>
                                    <br clear="all">
                                    <div><br>
                                    </div>
                                    -- <br>
                                    Best regards,<br>
                                    Roman. </div>
                                </blockquote>
                                <br>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                      <br>
                      <br clear="all">
                      <div><br>
                      </div>
                      -- <br>
                      Best regards,<br>
                      Roman. </div>
                  </blockquote>
                  <br>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <div><br>
        </div>
        -- <br>
        Best regards,<br>
        Roman.
      </div>
    </blockquote>
    <br>
  </div></div></div>

</blockquote></div><br><br clear="all"><div><br></div>-- <br>Best regards,<br>Roman.
</div>