<div dir="ltr">Hi,<div><br></div><div>We run a fairly large two server replica with 5 2TB bricks on each. We</div><div>recently added two more bricks each and started a rebalance.</div><div><br></div><div>We immediately started getting client errors and halted the rebalance.</div><div><br></div><div>Since then we've been getting more and more errors.</div><div><br></div><div>Symptoms:</div><div><br></div><div>Client: </div><div><br></div><div> gets Permission denied errors when accessing a file. If root, or</div><div> occasionally another user accesses the file, it works, then works for the</div><div> original user on that client.</div><div> <br></div><div> logs contain lots of:</div><div><br></div><div> [2014-10-30 03:14:08.912942] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-storage-client-3: remote operation failed: Permission denied. Path: /wms_pyramid/r_00000525/12285/r_00000525/8/8.qix (00000000-0000-0000-0000-000000000000)</div><div><br></div><div><div><br></div><div> errors.</div><div><br></div><div><br></div><div>Server:</div><div><br></div><div> link files and source files (that have caused problems above) seem to have different ownership:</div><div><br></div><div> $ ls -l brick*b/vol/raster/r_000002ff/000008ba.tif<br></div><div><div> ---------T 2 otm otm 0 Oct 30 17:03 brick0b/vol/raster/r_000002ff/000008ba.tif</div><div> -rw-rw-r-- 2 dandelion dandelion 15867977 Mar 22 2013 brick1b/vol/raster/r_000002ff/000008ba.tif</div><div><br></div><div><br></div><div> $ sudo getfattr -e hex -m- -d brick*b/vol/raster/r_000002ff/000008ba.tif</div><div> # file: brick0b/vol/raster/r_000002ff/000008ba.tif</div><div> trusted.afr.storage-client-0=0x000000000000000100000000</div><div> trusted.afr.storage-client-1=0x000000000000000100000000</div><div> trusted.gfid=0x54b3b2ae42504754a193505a933c30b7</div><div> trusted.glusterfs.dht.linkto=0x73746f726167652d7265706c69636174652d3100</div><div><br></div><div> # file: brick1b/vol/raster/r_000002ff/000008ba.tif</div><div> trusted.afr.storage-client-2=0x000000000000000000000000</div><div> trusted.afr.storage-client-3=0x000000000000000000000000</div><div> trusted.gfid=0x54b3b2ae42504754a193505a933c30b7</div></div><div><br></div><div><br></div><div> logs contain lots of:</div><div><br></div><div> [2014-10-29 19:43:35.838942] I [server-rpc-fops.c:575:server_mknod_cbk] 0-storage-server: 15505: MKNOD /media/settings/branding/mfe-logo-white-2-1.png (1776f2b8-8857-4801-8c58-266eafcd7a87/mfe-logo-white-2-1.png) ==> (Permission denied)</div><div><br></div><div> and</div><div><br></div><div> [2014-10-30 04:28:58.707352] E [marker.c:2080:marker_setattr_cbk] 0-storage-marker: Operation not permitted occurred during setattr of <nul></div><div>[2014-10-30 04:28:58.707407] I [server-rpc-fops.c:1778:server_setattr_cbk] 0-storage-server: 1210198: SETATTR /raster/r_0000081a/0000005e.jp2.aux.xml (0df0682b-0791-4327-bb5d-72ed916349fd) ==> (Operation not permitted)</div><div><br></div><div><br></div><div> This is happening to old files that haven't been changed since long before the rebalance, and is still happening, even though I believe the rebalance has been stopped.</div><div><br></div><div>We have restarted gluster-server on both servers. volume heal shows no current split-brain or heal-failed.</div><div><br></div><div><br></div><div> It does seem, to me, to match:</div><div> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=884597">https://bugzilla.redhat.com/show_bug.cgi?id=884597</a><br></div><div><br></div><div> Except that is supposedly fixed before 3.4.5.</div><div><br></div><div><br></div><div>Help!?</div><div><br></div><div><br></div><div>- Colin</div>-- </div><div><div dir="ltr">--<div>Colin Coghill</div><div>DevOps Engineer</div><div>Koordinates</div><div><a href="mailto:colin.coghill@koordinates.com" target="_blank">colin.coghill@koordinates.com</a></div></div>
</div></div>